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(54) Image signal data structure, image coding method, and image decoding method 



(57) A data structure of an image signal includes a 
reproduction cycle identifier indicating whether the cycle 
of image display for each frame is variable or not. When 
the display cycle identifier indicates that the display 
cycle is fixed, display cycle data is inserted in a header 
of the coded image data, and data relating to the frame 
number is inserted in each frame. On the other hand, 
when the display cycle identifier indicates that the dis- 
play cycle is variable, display time data is inserted for 



each frame. Therefore, when decoding and displaying 
the coded image data having a fixed display cycle, the 
decoded image data can be displayed by a simple cir- 
cuit structure, i.e., based on the display cycle data and 
the frame number data having a relatively small data 
quantity (bit number), without referring to the display 
time data having a relatively large data quantity for each 
frame. Furthermore, this data structure can be applied 
to a coded image signal having a variable display cycle. 



Fig.l (a) 



header H 



frame F(0) 



frame F(1) 



frame F<2) 



9 f^j coded [S3 coded ^ 

1 IS iroage data ^ image data ^ image data 



coded 



display cycle display frame number 
identifier cycle data data 

Of Dp 80 CgO 



I rams number 
data 
81 



Cgi 



frame number 
data 
B2 



Cg2 



frame F(n) 



^) coded 
^ image data 



1 



frame number 
data 
Bn Cgn 



100a 



SaO 



Sal 



Sa2 



San 



Fig.1 (b) 



display cycle 
identifier 
Of 



frame F(0) 



frame F< 1 ) 



frame F(2) 



coded 
image data 



coded 
image data 



coded 
Image data 



t 



disptay 
time data 
OtO CgO 

v v ' 

SbO 



f 



display display 
time data rime data 

Ot1 Cg1 Ot2 Cg2 



Sbl 



Sb2 



frame F(n) 



coded 
Image data 



display 
time data 

Dtn Cgn 

v ^- / 

Sbn 



100b 



Primed D/ Xerox (UK) Bus«ness Services 
2.16.7/3 6 



EP 0 914 008 A2 



Description 

FIFLD OF THE INVENTION 
[0001] 



01, Thepresentinvent.onr^^ 
ing method. More particularly, the invention ^^"^^^^yior each of frames as components of 

?^2££^ timin9 data ' and decodin9 of the 

n S^-Presentl^ 

ES ^f 9 .^ — - — 9 an ima9e process,ng pro ' 

g ™m ^^or implementing the above^escribed coding and decoding by us.ng a computer. 
,5 R^r.KPjROtlND ™ THF INVENTION 

t ^t =.r, a m. iltimedia in which audio, video and other data are integrally han- 
[0003] in recent years, we have greeted the age of ™'^^ a ^^f to me n such as newspapers, 

sary to convert the .nformafion «*°^^* 9 ^ZLrto«i above is estimated as a quantrty of digital data. 
[0004] When the data quantity of each mforma on med um d ^ case of h the dala quantity is 

in case of characters, the data quantty for ^f^^X^^^^ A is more than 100Mbits per second 
64Kbits per second (quality ^^^°^J 0 ^ ZZZuc* as televisions, it is not practica. to proc- 
(quality for current television ^ dc ^>^*V n ^'J^^to although visual telephones have already been put to 

sible to transmit an image of a ^^ZTalVL^ Tn'cafe of visual telephones, a moving picture compression 
[0005] So. data compressor Sector) is 

r P Ced 9 ^t™^ 

with audio data, in an ordinary music CD (compact d,sk > standard relating to a technology for compressing 

[0006] MPEG (Moving Picture Experts Group) ,s an ,nternat ona. ^^pQI is a sSr^S for compressing moving 
and expanding an image signa. oomjjond^ . f££2l the transmission rate to 

S5t.sk * ^ ^ ^ * 

2~ 1 5Mbps has been standardized to meet the demand to MPEG1 and M PEG2 which have 

[0007] in the image signal compressor . and ^^TZoy^Z interTaTs between image display tim- 
a.ready been put to P*^** ^^^ V^JStSL of frame rates, and in MPEG2 a frame rate 

z^ p :z\'™^ «« ** is se,ectai ,rom p,urai ,rame rates (frame 

rate values) with reference to a table shown ,n ^e™- proceeded by the working group for stand- 

[0008] Underthe :ex,st,n<; |«S^^ c '5gSS^") enables coding and signal operation in object 

a r a r e ^^^ 

p ressin 9 nc P ,uding high-bit-rate image processirvg ^^^^ f ^ 3) is added at the beginning of a 
[0009] Also in MPEG4, when a table similar to the tab e for MPE G2 (reteMo j ' according to the table. 

imaae disDlay timings or decoding timings of the respective frames. 
[0011 ] Rgure 14 shows a data structure of a convention^ coded ,mage signal 200. 
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[001 2] The coded image signal 200 corresponds to one image (in MPEG4, a series of frames constituting an image 
corresponding to one object) and includes a header H at the beginning. The header H is followed by code sequences 

SaO, Sa1, Sa2 San corresponding to frames F(0), F(1), F(2) F(n), respectively, which code sequences are 

arranged according to priority for transmission (transmission order). Here, "n M is the number indicating the transmission 
5 order of data of each frame in the frame sequence corresponding to one image, and n of the head frame is 0. 

[0013] In this example, at the beginnings of the code sequences SaO, Sa1, Sa2, .... San of the respective frames, 
display time data DtO, Dtl , Dt2, .... Dtn indicating the display timings of the frames are arranged. The respective display 
time data are followed by coded image data CgO, Cg1 , Cg2, .... Cgn. 

[001 4] Since each of the display time data indicates a time relative to a reference time, the quantity of data required 
io for expressing this display time, i.e., the bit number of the display time data, increases as the number of the frames con- 
stituting the image increases. 

[001 5] Further, at the decoding end of the coded image signal, according to the display time data DtO-Dtn inserted in 
the code sequences SaO-San corresponding to the respective frames, image display of each frame is carried out at the 
time indicated by the display time data. 

15 [001 6] Figure 15 shows the transmission order and the display order of the coded image data corresponding to each 
frame in the series of frames. As described above, "n" indicates the transmission order, and "n"' indicates the display 
order (n of the head frame is 0). Further, frames F(n) (F(0)~F(18)) are arranged based on the order of frames in the 
data structure shown in figure 14 (transmission order). The frames F(n) arranged in the transmission order are rear- 
ranged according to the display order of the frames as shown by arrows in figure 15, resulting in frames F'(n') 

20 (F'(0)~F*(18)) arranged in the display order. Accordingly, a frame F(n) and a frame F'(n') related to each other with an 
arrow are identical. For example, the frames F(0), F(1), F(2), and F(3) are identical to the frames F'(0), F'(3), F'(1), and 
F'(2) t respectively. 

[0017] Amongst the frames F(n) (F(0)~F(18)) arranged in the transmission order, the frames F(0) and F(13) are I 
(Intra-picture) frames (hereinafter also referred to as l-VOP), the frames F(1), F(4), F(7). F(10). and F(16) are P (Pre- 
25 dictive-picture) frames (hereinafter also referred to as P-VOP), and the frames F(2), F(3). F(5), F(6), F(8), F(9), F(11). 
F(12), F(14), F(15), F(17), and F(18) are B (Bidirectionally predictive picture) frames (hereinafter also referred to as B- 
VOP). 

[0018] When the frames F(n) (F(0)~F(18)) arranged in the transmission order (IPBBPBBPBBPBBIBBPBB) are rear- 
ranged in the display order (IBBPBBPBBPBBPBBIBBP), this display order n' is represented by frame numbers B(n) 
30 (B(0)~B(18)) corresponding to the respective frames F(n). That is, the frame numbers B(n) represent the numbers n' 
indicating the display order. To be specific, as shown in figure 15, B(0)=0, B(1)=3, .... B(17)=16, B(18)=17. Accordingly, 
the image display cycle L of the l-VOPs is 15, and the image display cycle M of the VOPs including both of the l-VOPs 
and the P-VOPs is 3. 

[0019] The frame number B(n)=n' is represented by the following formulae (1)~(3) using n. 

35 

B(n)=n=0 (n=0) (1) 

B(n)=n+M-1 (n=Mxi+1) (2) 

ao wherein i and M are integers not less than 0 (0, 1 , 2, ...). 

B(n)=n-1 (when n is other than the above values) (3) 

[0020] The first l-VOP satisfies the condition (n=0), the l-VOPs other than the first l-VOP and the P-VOPs satisfy the 
45 condition (n=Mxi+i ), and the B-VOPs satisfy the condition (when n is other than the above values). 

[0021] Formulae (l)-{3) define the relationship B(n)=n* between the display order n' and the transmission order n in 
the case where the code sequences of the frames corresponding to the respective l-VOPs, P-VOPs, and B-VOPs are 
transmitted periodically. In other cases than mentioned above, the display order n' and the transmission order n are cor- 
related one to one by a relational expression or a method other than formulae (1 )~(3). 
so [0022] Figure 1 6 is a diagram for explaining an example of an image display method wherein the intervals of the image 
display timings of the respective frames are variable. 

[0023] In the figure, t (n') (t (1), t (2), t'(3), t'(4),...) indicates the interval between the time at which image display of the 
frame F'(n'-1) is performed and the time at which image display of the frame F'(n') is performed, and h (1). h'(2). and 
h'(3) indicate the times for image display of the frames F'(1). F(2), and F'(3), respectively, with the time h*(0) for image 
55 display of the frame F(0) as a reference. Further. h(n) (h(1). h(2). h(3). h(4),...) indicates the time for image display of 
the frame F(n) (F(1), F(2), F(3), F(4),...) with the time h'(0) for image display of the frame F(0)=F'(0) as a reference. 
Accordingly, the display time rV(n') of the frame P(n') arranged in the display order is expressed by h , (n , )=h , (n , -1)+t , (n') , 
and h'(0)=0 . 
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.... .. Next, decodng and image display of the coded imago signal having the data ebocbtre shown in ligure u - 

be bri'efly described using ligora 16. „ »nn shown in fiaore 14 is input, the coded image data CgO. 

,0025] At the decoding end whan «eood^rnag' ^ ^I'^U signa, 200 are decoded. 

S™2?n the d£,a» lime da,a not OJ. DB - >%^°^ the .especbv. Irames „mage display 

and displayed at a prescribed t.mmg. respective frames of the coded image signal are 

rnrwn When the intervals between the image display timings ot tne respective ■ «» pm Ff2 . arec |j S - 

En^a^uS^^^ 
^ 0 9 rSbrha,,am.ra,a,e^ r .a,o,, = 

ir^ba^.aaa^nm^ 

numberl lor obtaining »".^™"'™ n =2 ^L^S, f SSl coda sequence Sbn corilains display cycle multiplier 

h,„. « ...) which is measured by ueing die s*u* 

fg^tX^^ -n-p-, .o .he respective bames TO 

iSr'S-er.Uen the image VOPO is an ,-VOP (b !^£E™£S^^£^:£S 

ures 14-16. ohiainpd bv codina an image signal having a fixed interval T of 

Ln image bavin, a ..»<*, bame rale, is ^^^ZSZS^XtSZ^**** by inereas- 
< s decoding end. ^^-'^.'^''SK len decoding the coded image eigne*, mere is no choir* 
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ures 17(a)-17(c). nrooosed bv the current MPEG4, even when the frame 

,„ simplify .he circuit structure forimp.ementing JV"^ 1 ^ 1 " 0 ,^^., , rame) . , he VO P, and the VOP2 are B- 
,0040) Thia problem will be bneh, ^described Mien me %^ the , rame F(0) responding to 

^r^TeStrSS - ™ me display liming d, fhe 
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following B-VOP (B frame), cannot be known until the frame F(2) corresponding to the B-VOP (B frame) is transmitted. 
SUMMARY OF THE INVENTION 

5 [0041] The present invention is made to solve the above-described problems and has for its object to provide an 
image signal data structure which enables a reproduction process including decoding and image display for a coded 
image signal having a fixed cycle of reproduction for each frame at the decoding end, such as a fixed frame rate (cycle 
of image display for each frame) by using a simple hardware structure, and also enables a reproduction process for a 
coded image signal having a variable cycle of reproduction for each frame, such as a variable frame rate. 

10 [0042] Another object of the present invention is to provide an cage coding method and an image coding apparatus 
capable of performing an image coding process which enables a reproduction process including decoding and image 
display for a coded image signal having a fixed cycle of reproduction for each frame at the decoding end, such as a fixed 
frame rate (cycle of image display for each frame) by using a simple hardware structure, and also enables a reproduc- 
tion process for a coded image signal having a variable cycle of reproduction for each frame, such as a variable frame 

is rate. 

[0043] Still another object of the present invention is to provide an image decoding method and an image decoding 
apparatus capable of performing an accurate reproduction process including decoding and image display for a coded 
image signal, according to whether the cycle of reproduction for each frame is variable or not. 

[0044] Yet another object of the present invention is to provide an image signal data structure which enables detection 
20 of the value of a frame rate or the like of a coded image signal having a fixed frame rate or the like before decoding each 
frame, and simplifies various hardware structures for implementing a reproduction process including decoding and 
image display. 

[0045] A further object of the present invention is to provide an image coding method and an image coding apparatus 
capable of detecting the value of a frame rate or the like of a coded image signal having a fixed frame rate or the like 
25 before decoding each frame, and performing a reproduction process including decoding and image display for the 
coded image signal by a simple hardware structure. 

[0046] A still further object of the present invention is to provide an image decoding method and an image decoding 

apparatus capable of accurately decoding the coded image signal obtained by the above -described coding process. 

[0047] Another object of the present invention is to provide a data storage medium containing a coded image signal 
30 having the above-described data structure, and a data storage medium containing an image processing program for 

implementing, with a computer, the above-described image coding method and image decoding method. 

[0048] Other objects and advantages of the invention will become apparent from the detailed description that follows. 

The detailed description and specific embodiments described are provided only for illustration since various additions 

and modifications within the spirit and scope of the invention will be apparent to those of skill in the art from the detailed 
35 description. 

[0049] According to a first aspect of the present invention, there is provided a data structure of an image signal for 
which image reproduction of each frame is performed at a prescribed cycle, and the image signal data structure 
includes a reproduction cycle identifier indicating whether the cycle of image reproduction for each frame is variable or 
not. Therefore, when the reproduction cycle for each frame is fixed, reproduction of the image signal, such as decoding 

40 of the coded image signal and display of the decoded image signal, can be performed by a simple circuit structure. 
[0050] According to a second aspect of the present invention, in the image signal data structure of the first aspect, 
the reproduction cycle identifier is a display cycle identifier indicating whether the cycle of image display for each frame 
is variable or not. Therefore, when the image display cycle for each frame is fixed, display of decoded data obtained by 
decoding the coded image signal can be performed by a simple circuit structure. 

45 [0051 ] According to a third aspect of the present invention, the image signal data structure of the second aspect 
includes a fixed display cycle identifier indicating that the cycle of image display for each frame is fixed, as the display 
cycle identifier; display cycle data indicating the cycle of image display for each frame; and frame position data corre- 
sponding to each frame and indicating the positional relationship of each frame with the previous and subsequent 
frames. Therefore, when the image display cycle for each frame is fixed, the timing of image display of the frame can 

so be defined according to the display cycle data and the frame position data. 

[0052] According to a fourth aspect of the present invention, the image signal data structure of the second aspect 
includes a variable display cycle identifier indicating that the cycle of image display for each frame is variable, as the 
display cycle identifier; and display timing data indicating the timing at which image display of each frame is performed, 
the timing being set relatively to a desired reference time selected from at least one reference time according to each 

55 frame. Therefore, when the image display cycle for each frame is variable, the timing of image display of the frame can 
be defined according to the display timing data. 

[0053] According to a fifth aspect of the present invention, in the image signal data structure of the first aspect, the 
reproduction cycle identifier is a decoding cycle identifier indicating whether the cycle of decoding of a coded image sig- 
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nal corresponding to each frame is variable or not. Therefore, when the decoding cycle for each frame is fixed, decoding 
of the coded image signal can be performed by a simple circuit structure. 

{0054] According to a sixth aspect of the present invention, the image signal data structure of the fifth aspect includes 
a fixed decoding cycle identifier indicating that the cycle of decoding for each frame is fixed, as the decoding cycle iden- 
5 tifier; decoding cycle data indicating the cycle of decoding for each frame; and frame position data corresponding to 
each frame and indicating the positional relationship of each frame with the previous and subsequent frames. There- 
fore, when the decoding cycle for each frame is fixed, the timing of decoding of the frame can be defined according to 
the decoding cycle data and the frame position data. 

[0055] According to a seventh aspect of the present invention, the image signal data structure of the fifth aspect 

w includes a variable decoding cycle identifier indicating that the cycle of decoding for each frame is variable, as the 
decoding cycle identifier; and decoding timing data indicating the timing at which decoding of each frame is performed, 
the timing being set relatively to a desired reference time selected from at least one reference time, according to each 
frame. Therefore, when the decoding cycle for each frame is variable, the timing of decoding of the frame can be 
defined according to the decoding timing data. 

is [0056] According to an eighth aspect of the present invention, there is provided an image coding method for coding 
an image signal corresponding to a specific image to generate a coded image signal, and outputting the coded image 
signal together with a reproduction cycle identifier indicating whether the cycle of image reproduction for each of frames 
constituting the image is variable or not. In this method, when an image signal having a fixed cycle of image reproduc- 
tion for each frame is input as the image signal to be coded, reproduction cycle data and frame position data are output 

20 together with the reproduction cycle identifier, the reproduction cycle data indicating the cycle of image reproduction for 
each frame, and the frame position data corresponding to each frame and indicating the positional relationship of each 
frame with the previous and subsequent frames. When an image signal having a variable cycle of image reproduction 
for each frame is input as the image signal to be coded, reproduction timing data is output together with the reproduc- 
tion cycle identifier, the reproduction timing data indicating the timing at which image reproduction of each frame is per- 

25 formed, the timing being set relatively to a desired reference time selected from at least one reference time, according 
to each frame. In this method, an image signal having a variable cycle of image reproduction for each frame and an 
image signal having a fixed cycle of image reproduction for each frame can be coded together with data indicating the 
reproduction timing of each frame according to each image signal. Therefore, when the reproduction cycle for each 
frame is fixed, the bit number required for setting the reproduction time can be reduced and, furthermore, reproduction 

30 of the image having a variable frame reproduction cycle can be performed in the same manner as the conventional 
reproduction. 

[0057] According to a ninth aspect of the present invention, in the image coding method of the eighth aspect, the 
reproduction cycle identifier is a display cycle identifier indicating whether the cycle of image display for each frame is 
variable or not; the reproduction cycle data is display cycle data indicating the cycle of image display for each frame; 

35 and the reproduction timing data is display timing data indicating the timing at which image display of each frame is per- 
formed, the timing being set relatively to a desired reference time selected from at least one reference time, according 
to each frame. Therefore, when the display cycle for each frame is fixed, the bit number required for setting the display 
time can be reduced and, furthermore, display of the image having a variable frame display cycle can be performed in 
the same manner as the conventional display. 

40 [0058] According to a tenth aspect of the present invention, in the image coding method of the eighth aspect, the 
reproduction cycle identifier is a decoding cycle identifier indicating whether the cycle of decoding of coded image data 
corresponding to each frame is variable or not; the reproduction cycle data is decoding cycle data indicating the cycle 
of decoding for each frame; and the reproduction timing data is decoding timing data indicating the timing at which 
decoding of each frame is performed, the timing being set relatively to a desired reference time selected from at least 

45 one reference time, according to each frame. Thereby, when the decoding cycle for each frame is fixed, the bit number 
required for setting the decoding time of each frame can be reduced and, furthermore, decoding of the image having a 
variable frame decoding cycle can be performed in the same manner as the conventional decoding. 
[0059] According to an eleventh aspect of the present invention, there is provided an image decoding method for 
decoding a coded image signal which includes coded image data corresponding to frames constituting an image, and 

so a reproduction cycle identifier indicating whether the cycle of image reproduction for each frame is variable or not. In 
this method, when the reproduction cycle identifier indicates that the cycle of image reproduction for each frame is fixed, 
decoded image data obtained by decoding the coded image data corresponding to each frame becomes image data 
having a reproduction timing which is decided according to reproduction cycle data indicating the cycle ol image repro- 
duction for each frame and frame position data indicating the positional relationship of each frame with the previous and 

55 subsequent frames, which data are included in the coded image signal. When the reproduction cycle identifier indicates 
that the cycle of image reproduction for each frame is variable, decoded image data obtained by decoding the coded 
image data corresponding to each frame is image data having a reproduction timing decided according to reproduction 
timing data which is included in the coded image signal and indicates the timing at which image reproduction of each 



6 



EP 0 914 008 A2 

frame is carried out, the timing being set relatively to a desired reference time selected from at least one reference time, 
according to each frame. Therefore, reproduction of the coded image signal, including decoding and display, can be 
accurately performed according to whether the reproduction timing of each frame of the coded image signal is variable 
or not. 

f [0060] According to a twelfth aspect of the present invention, in the image decoding method of the eleventh aspect, 
the reproduction cycle identifier is a display cycle identifier indicating whether the cycle of image display for each frame 
is variable or not; the reproduction cycle data is display cycle data indicating the cycle of image display for each frame; 
and the reproduction timing data is display timing data indicating the timing at which image display of each frame is per- 
formed. Therefore, display of the image signal can be accurately performed according to whether the display timing of 

w each frame of the coded image signal is variable or not. 

[0061] According to a thirteenth aspect of the present invention, in the image decoding method of the eleventh aspect, 
a decoding timing at which decoding of each frame is carried out is set according to display timing data of plural frames 
including an object frame to be decoded. Therefore, not only display but also decoding of each frame can be carried out 
according to the display timing data. 

is [0062] According to a fourteenth aspect of the present invention, in the image decoding method of the thirteenth 
aspect, based on the display timing data of the object frame and the display timing data of the next frame transmitted 
subsequently to the object frame, the decoding timing of the object frame is set at a timing that is earlier, by a prescribed 
offset time, than the earlier display timing between the display timings of the object frame and the next frame. Therefore, 
decoding of each frame can be performed according to the display timing data, without impeding the flow of display. 

20 [0063] According to a fifteenth aspect of the present invention, in the image decoding method of the fourteenth aspect, 
when the display timing of the object frame is earlier than the display timing of the next frame, the offset time is set to a 
length longer than the time required for decoding the object frame; and when the display timing of the next frame is ear- 
lier than the display timing of the object frame, the offset time is set to a length longer than the sum of the time required 
for decoding the object frame and the time required for decoding the next frame. Therefore, decoding of a coded image 

25 data corresponding to a series of frames including P-VOP and B-VOP as well as l-VOP can be performed according to 
the display timing data. 

[0064] According to a sixteenth aspect of the present invention, in the image decoding method of the eleventh aspect, 
the reproduction cycle identifier is a decoding cycle identifier indicating whether the cycle of decoding of coded image 
data corresponding to each frame is variable or not; the reproduction cycle data is decoding cycle data indicating the 

30 cycle of decoding for each frame; and the reproduction timing data is decoding timing data indicating the timing at which 
decoding of each frame is performed. Therefore, decoding of the coded image signal can be accurately performed 
according to whether the decoding timing of each frame of the coded image signal is variable or not. 
[0065] According to a seventeenth aspect of the present invention, the image signal data structure of the first aspect 
further includes sub-unit time data indicating the length of a sub-unit time which is obtained by dividing a prescribed 

35 time interval into N (natural number), by the natural number N; and reproduction cycle multiplier data indicating that the 
cycle of image reproduction for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. 
Therefore, when a coded image signal having a fixed frame rate is processed, the value of the frame rate can be 
detected before decoding each frame, whereby various hardware structures for implementing reproduction including 
decoding and display can be simplified. 

40 [0066] According to an eighteenth aspect of the present invention, in the image signal data structure of the seven- 
teenth aspect, the reproduction cycle identifier is a display cycle identifier indicating whether the cycle of image display 
for each frame is variable or not; and the reproduction cycle multiplier data is display cycle multiplier data which indi- 
cates that the cycle of image display for each frame is the sub-unit time multiplied by M (natural number), by the multi- 
plier M. Therefore, various hardware structures for implementing image display can be simplified. 

45 [0067] According to a nineteenth aspect of the present invention, in the image signal data structure of the seventeenth 
aspect, the reproduction cycle identifier is a decoding cycle identifier indicating whether the cycle of decoding for each 
frame is variable or not; and the reproduction cycle multiplier data is decoding cycle multiplier data which indicates that 
the cycle of decoding for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. Therefore, 
various hardware structures for implementing decoding can be simplified. 

50 [0068] According to a twentieth aspect of the present invention, there is provided an image coding method for coding 
an image signal corresponding to a specific image to generate a coded image signal, and outputting the coded image 
signal together with a reproduction cycle identifier indicating whether the cycle of image reproduction for each of frames 
constituting the image is variable or not. In this method, when an image signal having a fixed cycle of image reproduc- 
tion for each frame is input as the image signal to be coded, sub-unit time data and reproduction cycle multiplier data 

55 are output together with the reproduction cycle identifier, the sub-unit time data indicating the length of a sub-unit time 
which is obtained by dividing a prescribed time interval into N (natural number), by the natural number N. and the repro- 
duction cycle multiplier data indicating that the cycle of image reproduction for each frame is the sub-unit time multiplied 
by M (natural number), by the multiplier M. Therefore, when a coded image signal having a fixed frame rate is proc- 
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essed. the value of the frame rate can be detected before decoding each frame, whereby reproduction including decod- 
ing and display can be performed with a simple hardware structure. 

[0069] According to a twenty-first aspect of the present invention, in the image coding method of the twentieth aspect, 
the reproduction cycle identifier is a display cycle identifier indicating whether the cycle of image display for each frame 
5 is variable or not; and the reproduction cycle multiplier data is display cycle multiplier data which indicates that the cycle 
of image display for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. Therefore, var- 
ious hardware structures for implementing display at the decoding end can be simplified. 

[0070] According to a twenty-second aspect of the present invention, in the image coding method of the twentieth 
aspect the reproduction cycle identifier is a decoding cycle identifier indicating whether the cycle of decoding for each 
io frame is variable or not; and the reproduction cycle multiplier data is decoding cycle multiplier data which indicates that 
the cycle of decoding for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. Therefore, 
various hardware structures for implementing decoding can be simplified. 

[0071] According to a twenty-third aspect of the present invention, there is provided an image decoding method for 
decoding a coded image signal including the following data: coded image data corresponding to frames constituting an 
75 image; a reproduction cycle identifier indicating whether the cycle of image reproduction for each frame is variable or 
not; sub-unit time data indicating the length of a sub-unit time which is obtained by dividing a prescribed time interval 
into N (natural number), by the natural number N; and reproduction cycle multiplier data indicating that the cycle of 
image reproduction for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. In this 
method, when the reproduction cycle identifier indicates that the cycle ol image reproduction for each frame is fixed, 
20 decoded image data obtained by decoding the coded image data corresponding to each frame makes image data hav- 
ing a reproduction timing which is decided according to the sub-unit time data indicating the length of the sub-unit time 
and .the reproduction cycle multiplier data indicating the cycle of image reproduction for each frame. Therefore, when a 
coded image signal having a fixed frame rate is processed, the value of the frame rate can be detected before decoding 
each frame, whereby the hardware structure at the decoding end can be simplified. 

[0072] According to a twenty-fourth aspect of the present invention, in the image decoding method of the twenty-third 
aspect, the reproduction cycle identifier is a display cycle identifier indicating whether the cycle of image display for 
each frame is variable or not; and the reproduction cycle multiplier data is display cycle multiplier data which indicates 
that the cycle of image display for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. 
Therefore, the hardware structure for display at the decoding end can be simplified. 

[0073] According to a twenty-fifth aspect of the present invention, in the image decoding method of the twenty-third 
aspect the reproduction cycle identifier is a decoding cycle identifier indicating whether the cycle of decoding for each 
frame is variable or not; and the reproduction cycle multiplier data is decoding cycle multiplier data which indicates that 
the cycle of decoding for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. Therefore, 
the hardware structure for decoding at the decoding end can be simplified. 
35 [0074] According to a twenty-sixth aspect of the present invention, there is provided an image coding apparatus for 
coding an image signal corresponding to a specific image, and the apparatus comprises an encoder for coding an input 
image signal and outputting coded image data; a cycle decision unit for deciding whether the cycle of image reproduc- 
tion for each of frames constituting the image is variable or not, based on the image signal, and outputting a reproduc- 
tion cycle identifier indicating the result of the decision; a first data generator for generating reproduction cycle data 
40 indicating the cycle of image reproduction for each frame, according to the image signal; a second data generator for 
generating frame position data corresponding to each frame and indicating the positional relationship of each frame 
with the previous and subsequent frames, according to the image signal; a third data generator for generating repro- 
duction timing data indicating the timing at which image reproduction for each frame is carried out, according to the 
image signal; an ON/OFF switch for switching the circuit between the ON state where the reproduction cycle data is 
45 transmitted and the OFF state where the reproduction cycle data is cut off, according to the reproduction cycle identifier; 
a selector switch for selecting one of the frame position data and the reproduction timing data, according to the repro- 
duction cycle identifier; a multiplexer for multiplexing the outputs from the encoder, the cycle decision unit, and the 
switches in a prescribed order; and the image coding apparatus outputting a bit stream obtained by the multiplication 
as a coded image signal. In this apparatus, an image signal having a variable cycle of image reproduction for each 
so frame and an image signal having a fixed cycle of image reproduction for each frame can be coded with data indicating 
the reproduction timing of each frame according to each image signal. Therefore, when the reproduction cycle for each 
frame is fixed, the bit number required for setting the reproduction time can be reduced and, furthermore, reproduction 
of the image having a variable frame reproduction cycle can be performed easily. 

[0075] According to a twenty-seventh aspect of the present invention, there is provided an image decoding apparatus 
55 for decoding and reproducing the coded image signal output from the image coding apparatus of the twenty-sixth 
aspect, and the apparatus comprises a demultiplexer for receiving the coded image signal, and separating the coded 
image signal into the coded image data, the reproduction cycle identifier, the reproduction cycle data, the frame position 
data, and the reproduction timing data, and then outputting these data; a decoder for decoding the coded image data, 
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frame by frame, to generate decoded image data; an ON/OFF switch for switching the circuit between the ON state 
where the reproduction cycle data is transmitted and the OFF state where the reproduction cycle data is cut off, accord- 
ing to the reproduction cycle identifier; a selector switch for selecting one of the frame position data and the reproduc- 
tion timing data, according to the reproduction cycle identifier; and a display unit for performing image display of each 

5 frame according to the decoded image data; wherein at least one of the decoding by the decoder and the image display 
by the display unit is performed at a reproduction timing decided by the reproduction cycle data and the frame position 
data or a reproduction timing decided by the reproduction timing data, according to the reproduction cycle identifier. 
Therefore, reproduction of the coded image signal, including decoding and display, can be accurately performed 
according to whether the reproduction timing of each frame of the coded image signal is variable or not. 

10 [0076] According to a twenty-eighth aspect of the present invention, there is provided an image coding apparatus for 
coding an image signal corresponding to a specific image, and the apparatus comprises an encoder for coding an input 
image signal and outputting coded image data; a cycle decision unit for deciding whether the cycle of image reproduc- 
tion for each of frames constituting the image is variable or not, based on the image signal, and outputting a reproduc- 
tion cycle identifier indicating the result of the decision; a first data generator for generating sub-unit time data indicating 

15 the length of a sub-unit time which is obtained by dividing a prescribed time interval into N (natural number), by the nat- 
ural number N; a second data generator for generating reproduction cycle multiplier data indicating that the cycle of 
image reproduction for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. according 
to the image signal; a third data generator for generating reproduction timing data indicating the timing at which image 
reproduction of each frame is carried out, according to the image signal; a first ON/OFF switch for switching the circuit 

20 between the ON state where the sub-unit time data is transmitted and the OFF state where the sub-unit time data is cut 
off. according to the reproduction cycle identifier; a second ON/OFF switch for switching the circuit between the ON 
state where the reproduction cycle multiplier data is transmitted and the OFF state where the reproduction cycle multi- 
plier data is cut off, according to the reproduction cycle identifier; a multiplexer for multiplexing the outputs from the 
encoder, the cycle decision unit, the third data generator, and the first and second ON/OFF switches in a prescribed 

25 order; and the image coding apparatus outputting a bit stream obtained by the multiplication as a coded image signal. 
Therefore, when a coded image signal having a fixed frame rate is processed at the decoding end. the value of the 
frame rate can be detected before decoding each frame, whereby reproduction of the coded image signal, including 
decoding and display, can be performed by a simple hardware structure. 

[0077] According to a twenty-ninth aspect of the present invention, there is provided an image decoding apparatus 
30 for decoding and reproducing the coded image signal output from the image coding apparatus of the twenty-eighth 
aspect, and the apparatus comprises a demultiplexer for receiving the coded image signal, and separating the coded 
image signal into the coded image data, the reproduction cycle identifier, the sub-unit time data, the reproduction cycle 
multiplier data, and the reproduction timing data, and then outputting these data; a decoder for decoding the coded 
image data, frame by frame, to generate decoded image data; a first ON/OFF switch for switching the circuit between 
35 the ON state where the reproduction cycle multiplier data is transmitted and the OFF state where this data is cut off. 
according to the reproduction cycle identifier; a second ON/OFF switch for switching the circuit between the ON state 
where the reproduction timing data is transmitted and the OFF state where this data is cut off, according to the repro- 
duction cycle identifier; a display unit for performing image display of each frame according to the decoded image data; 
wherein at least one of the decoding by the decoder and the image display by the display unit is performed at a repro- 
40 duction timing decided by the sub-unit time data and the reproduction cycle multiplier data or a reproduction timing 
decided by the reproduction timing data, according to the reproduction cycle identifier. Therefore, when a coded image 
signal having a fixed frame rate is processed, the value of the frame rate can be detected before decoding each frame, 
whereby the hardware structure at the decoding end can be simplified. 

[0078] According to a thirtieth aspect of the present invention, there is provided a data storage medium containing an 
45 image signal for which image reproduction for each frame is performed at a prescribed cycle, and the image signal 
includes a reproduction cycle identifier indicating whether the cycle of image reproduction for each frame is variable or 
not. Therefore, by using the data storage medium, when the reproduction cycle for each frame is fixed, reproduction of 
the image signal, such as decoding of the coded image signal and display of the decoded image signal, can be per- 
formed with a simple circuit structure. 
so [0079] According to a thirty-first aspect of the present invention, there is provided a data storage medium containing 
an image signal for which image reproduction for each frame is performed at a prescribed cycle, and the image signal 
includes a reproduction cycle identifier indicating whether image reproduction for each frame is variable or not; sub-unit 
time data indicating the length of a sub-unit time which is obtained by dividing a prescribed time interval into N (natural 
number), by the natural number N; and reproduction cycle multiplier data indicating that the cycle of image reproduction 
55 for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. Therefore, by using the data 
storage medium, the value of a fixed frame rate of a coded image signal can be detected before decoding each frame, 
whereby various hardware structures for implementing reproduction including decoding and display can be simplified. 
[0080] According to a thirty-second aspect of the present invention, there is provided a data storage medium contatn- 
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ing an image processing program, and the image processing program is a coding program which enables a computer 
to execute the coding of an image signal by the image coding method of the eighth aspect. Therefore, by loading the 
program in a computer, it is possible to implement an apparatus that can reduce the bit number required for setting the 
reproduction time when the frame reproduction cycle is fixed, and that can facilitate reproduction of an image having a 
variable frame reproduction cycle. 

[0081] According to a thirty-third aspect of the present invention, there is provided a data storage medium containing 
an image processing program, and the image processing program is a decoding program which enables a computer to 
execute the decoding of a coded image signal by the image decoding method of the eleventh aspect. Therefore, by 
loading the program in a computer, it is possible to implement an apparatus that can accurately perform reproduction 
of the coded image signal, including decoding and display, according to whether the reproduction timing of each frame 
is variable or not. 

[0082] According to a thirty-fourth aspect of the present invention, there is provided a data storage medium containing 
an image processing program, and the image processing program is a coding program which enables a computer to 
execute the coding of an image signal by the image coding method of the twentieth aspect. Therefore, by loading the 
program in a computer, it is possible to implement an apparatus that can detect the value of a fixed frame rate of a 
coded image signal before decoding each frame, and that can perform reproduction of the coded image signal, includ- 
ing decoding and display, by a simple hardware structure. 

[0083] According to a thirty-fifth aspect of the present invention, there is provided a data storage medium containing 
an image processing program, and the image processing program is a decoding program which enables a computer to 
execute the decoding of a coded image signal by the image decoding method of the twenty-third. Therefore, by loading 
the program in a computer, it is possible to implement an apparatus that can detect the value of a fixed frame rate of a 
coded image signal before decoding each frame, and that can simplify the hardware structure at the decoding end. 

BRIEF DESCRIPTION OF THE DRAWINGS 



[0084] 

Figures 1 (a) and 1 (b) are diagrams illustrating the data structure of a coded image signal having a fixed frame dis- 
play cycle and the data structure of a coded image signal having a variable frame display cycle, respectively, 
according to a first embodiment of the present invention. 

Figure 2 is a diagram for explaining image display based on the coded image signal having a fixed frame display 
cycle according to the first embodiment. 

Figure 3 is a flowchart of a coding process for generating the coded image signals having the data structures 
according to the first embodiment. 

Figures 4(a) and 4(b) are block diagrams illustrating the structures of image coding apparatuses according to the 
first embodiment and a modification of the first embodiment, respectively. 

Figure 5 is a flowchart of a decoding process for decoding the coded image signals having the data structures 
according to the first embodiment. 

Figures 6(a) and 6(b) are block diagrams illustrating the structures of image decoding apparatuses according to the 
first embodiment and the modification thereof, respectively. 

Figures 7(a) and 7(b) are diagrams illustrating the data structure of a coded image signal based on MPEG4 and 
having a fixed frame display cycle, and the data structure of a coded image signal based on MPEG4 and having a 
variable frame display cycle, respectively, according to a second embodiment of the present invention. 
Figure 8 is a flowchart of a coding process for generating the coded image signals having the data structures 
according to the second embodiment. 

Figures 9(a) and 9(b) are block diagrams illustrating the structures of image coding apparatuses according to the 
second embodiment and a modification of the second embodiment, respectively. 

Figure 10 is a flowchart of a decoding process for decoding the coded image signals having the data structures 
according to the second embodiment. 

Figures 1 1 (a) and 1 1 (b) are block diagrams illustrating the structures of image decoding apparatuses according to 
the second embodiment and the modification thereof, respectively. 

Figures 12(a) and 12(b) are diagrams for explaining a data storage medium containing a program for executing the 
coding or decoding process according to any of the aforementioned embodiments by a computer system, and fig- 
ure 12(c) is a diagram for explaining the computer system. 
Figure 13 is a diagram illustrating a table of fixed frame rates of MPEG2. 

Figure 14 is a diagram for explaining a coded image signal having a conventional data structure. 
Figure 15 is a diagram illustrating the data transmission order in a series of frames constituting one image, in com- 
parison with the data display order. 
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Figure 16 is a diagram for explaining image display based on a conventional coded image signal having a variable 
frame display cycle. 

Figures 1 7(a) and 17(b) are diagrams for explaining a method for expressing display times of frames (VOPs) based 
on MPEG4, and figure 17(c) is a diagram for explaining the current data structure of a coded image signal based 
5 on MPEG4. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[Embodiment 1] 

10 

[0085] Figure 1 (a) shows a data structure of a coded image signal 1 00a having a fixed (constant) frame display cycle 
according to a first embodiment of the present invention. Figure 1(b) shows a data structure of a coded image signal 
1 00b having a variable frame display cycle according to the first embodiment. 

[0086] The coded image signal 1 00a shown in figure 1 (a) is obtained by coding an image signal corresponding to one 
is image (in MPEG4, an image corresponding to one object) and having a fixed frame display cycle. The coded image sig- 
nal 100a includes a header H at the beginning, and the header H is followed by code sequences SaO, Sal , Sa2, .... San 
corresponding to the respective frames F(0). F(1), F(2) p .... F(n), which code sequences are arranged in the transmis- 
sion order. 

[0087] In the coded image signal 100a, the header H contains a display cycle identifier Df indicating that the frame 
20 display cycle is fixed (fixed display cycle identifier), and display cycle data Dp indicating the frame display cycle. Frame 
number data B0, B1 , B2 Bn are inserted at the beginnings of the code sequences SaO, Sa1 , Sa2,..., San correspond- 
ing to the respective frames, and each frame number data indicates the frame number B(n) corresponding to the order 

n' for displaying the frame. Further, the code sequences SaO, Sa1, Sa2 San corresponding to the respective frames 

contain coded image data CgO, Cg1, Cg2 Cgn obtained by coding image signals of the respective frames, respec- 
ts tively. 

[0088] Figure 2 is a diagram for explaining an example of an image display method in which the intervals of image 
display timings of the respective frames are fixed. In the figure, the same reference numerals as those shown in figure 
16 denote the same components, and T indicates a frame display cycle of an image in which the intervals of frame dis- 
play timings are fixed. 

30 [0089] In the coded image signal 100a, as shown in figure 2, the display times h(n) of the frames F(n) (n=0.1 .2,...) 
arranged in the transmission order are expressed by h(n)=B(n)xT when the display time h(0) (=h'(0)) of the frame F(0) 
is 0. To be specific, the display time h(2) of the frame F(2) is expressed by h(2)=B(2)xT , the display time h(3) of the 
frame F(3) is expressed by h(3)=B(3)xT , the display time h(1) of the frame F(1) is expressed by h(1)=B(1)xT , and the 
display time h(4) of the frame F(4) is expressed by h(4)=B(4)xT . 

35 [0090] Accordingly, in the reproduction of the coded image signal 100a. decoded image data obtained by decoding 
the coded image data corresponding to the respective frames are successively displayed at the display times h(n). The 
frame number B(n) indicating the number n' of the display order is determined as a function of the number n of the trans- 
mission order, according to the formulae (1)~(3). 

[0091] On the other hand, the coded image signal 100b shown in figure 1(b) is obtained by coding an image signal 
40 corresponding to one image (in MPEG4, an image corresponding to one object) and having a variable frame display 
cycle. The coded image signal 100b includes a header H at the beginning, and the header H is followed by code 

sequences SbO. Sb1 . Sb2 Sbn corresponding to the respective frames F(0), F(1). F(2) F(n) which are arranged 

in the transmission order. 

[0092] In the coded image signal 100b. the header H contains a display cycle identifier Df indicating that the frame 
45 display cycle is variable (variable display cycle identifier). Display time data (display timing data) DtO, Dt1 , Dt2, .... Dtn 

indicating the display times h(0). h(1), h(2) h(n) at which the respective frames F(0). F(1), F(2), .... F(n) are to be 

displayed are inserted at the beginnings of the code sequences SbO. Sb1 , Sb2 Sbn corresponding to the respective 

frames. Further, the coded sequences SbO. Sbl, Sb2 Sbn corresponding to the respective frames include coded 

image data CgO, Cg1 . Cg2 Cgn obtained by coding the image signals of the respective frames. 

so [0093] When reproducing the coded image signal 100b, image display is carried out in like manner as that described 
for the coded image signal 200 having the conventional data structure shown in figure 14. 
[0094] A description is now given of the function and effect. 

[0095] According to the first embodiment of the invention, in the coded image signal 1 00a having a fixed frame display 
cycle (figure 1(a)). the fixed display cycle identifier Df indicating that the frame display cycle is fixed and the display 
55 cycle data Dp indicating the frame display cycle are inserted in the header of the whole image data, and the frame 

number data B0. B1, B2, .... Bn respectively indicating the frame numbers B(0). B(1), B(2) B(n) are inserted in the 

respective frames. 

[0096] In the coded image signal 100a having such data structure, the display cycle data Dp indicates the frame dis- 
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play cycle T, and the frame number data Bn of each frame indicates what number the frame in one image is when 
counted in the display order. Therefore, the display time h(n) of each frame F(n) can be uniquely decided by these data 
Dp and Bn. 

[0097] On the other hand, in the coded image signal 1 00b having a variable frame display cycle (figure 1 (b)), the var- 
5 iable display cycle identifier Df indicating that the frame display interval is variable is inserted in the header of the whole 

image data, and the display time data DtO, Dt1 , Dt2, .... Dtn indicating the display times h(0), h(1), h(2) h(n) of the 

respective frames are inserted in the respective frames like the data structure of the conventional coded image signal 
200. 

[0098] Therefore, when reproducing the coded image signal 100b, the images of the respective frames F(0)-~F(n) are 

io displayed at the display times h(0)~h(n) indicated by the display time data Dt0~Dtn. 

[0099] As described above, since the display cycle identifier Df indicating whether the frame display cycle is fixed or 
variable is inserted in the header of the coded image signal, even an image having a variable frame display cycle can 
be handled. For an image having a fixed frame display cycle, image display of each frame can be carried out according 
to the display cycle data Dp and the frame number data Bn having a relatively small data quantity, without referring to 

75 the display time data DtO- Dtn having relatively large data quantities. Therefore, the circuit structure of an image 
processing circuit at the decoding end can be simplified. 

[0100] Hereinafter, a description is given of an image signal coding process for generating the coded image signals 
100a and 100b, and a decoding process for decoding these coded image signals. Figure 3 is a flowchart of the coding 
process. In the figure, "n" is the number corresponding to the transmission order of data of each frame in a series of 

20 frames constituting an image. 

[0101] In the coding process, first of all, it is decided whether the frame display cycle of an input image signal corre- 
sponding to a specific image is fixed or variable (step S1 1). When the result of the decision is that the frame display 
-. cycle is fixed, a fixed display cycle identifier Df indicating that the frame display cycle is fixed is added to the header H 
of the bit stream corresponding to the image signal (step S1 1 a). Further, the number n indicating the transmission order 

25 of each frame is used as a count value, and this count value n is set to 0 (step S1 2a). Subsequently, display cycle data 
Dp indicating the fixed frame display cycle T is added to the header of the bit stream corresponding to the image signal 
(step SI 3a). Next, as a code sequence SaO corresponding to the first frame F(0) of the specific image in the transmis- 
sion order, frame number data Bn(=B0) and coded image data Cgn(=Cg0) are successively added to the header H 
(steps S14a and SI 5a). Thereafter, it is decided whether or not the frame being processed in the image signal coding 

30 process (hereinafter referred to as "object frame") is the last frame of the specific image in the transmission order (step 
S16a). When the object frame is not the last frame, the count value n is incremented by 1 (step S1 7a), and the following 
frame F(1) is subjected to the processes of steps Sl4a~S17a. 

[0102] The processes of steps Sl4a~Sl7a are repeated until it is decided in step S16a that the object frame is the 
last frame. 

35 [0103] On the other hand, when the result of the decision in step S1 1 is that the frame display cycle is variable, a 
variable display cycle identifier of indicating that the frame display cycle is variable is added to the header H of the bit 
stream corresponding to the image signal (step Si 1b). Further, the number n indicating the transmission order of each 
frame is used as a count value, and this count value n is set to 0 (step S12b). Next, as a code sequence corresponding 
to the first frame F(0) of the specific image in the transmission order, display time data Dtn (=Dt0) and coded image data 

40 Cgn(=Cg0) are successively added to the header H (steps Sl3b and Si 4b). Thereafter, it is decided whether or not the 
frame being processed in the image signal coding process (object frame) is the last frame of the specific image in the 
transmission order (step S15b). When the object frame is not the last frame, the count value n is incremented by 1 (step 
Sl6b), and the following frame F(1) is subjected to the processes of steps Sl3b~Sl6b. 

[0104] The processes in steps S13b~Sl6b are repeated until it is decided in step Sl5b that the object frame is the 
45 last frame. 

[0105] Figure 4(a) is a block diagram illustrating an image coding apparatus 1 000 as hardware performing the coding 
process according to the first embodiment. 

[0106] The image coding apparatus 1000 includes an encoder 1 1 10 for coding an input image signal Sg to generate 
coded image data Cgn; a decision unit 1 131 for deciding whether the frame display cycle of the input image signal Sg 
so is fixed or not, i.e., fixed or variable, and outputting a display cycle identifier Df indicating whether the display cycle is 
fixed or not; and a display cycle data generator (first data generator) 1 1 32 for generating display cycle data Df indicating 
a fixed frame display cycle T based on the input image signal Sg. 

[0107] Further, the image coding apparatus 1000 includes a number data generator (second data generator) 1 1 33 for 
generating frame number data Bn indicating the number of each frame in the transmission order (frame number B(n)), 
55 based on the input image signal Sg; and a display time data generator (third data generator) 1 1 34 for generating display 
time data Dtn indicating the display time h(n) of each frame F(n), based on the input image signal Sg. 
[0108] Furthermore, the image coding apparatus 1000 includes an ON/OFF switch 1141 for switching the circuit 
between the ON state where the display cycle data Dp from the data generator 1 132 is transmitted and the OFF state 
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where the display cycle data Dp is cut off, based on the display cycle identifier Df from the decision unit 1131; and a 
selector switch 1 142 for selecting one of the frame number data Bn from the data generator 1 133 and the display time 
data Dtn from the data generator 1 134, based on the display cycle identifier Df from the decision unit 1131. 
[0109] Moreover, the image coding apparatus 1000 includes a multiplexer (MUX) 1120 for multiplexing the display 

£ cycle identifier Df from the decision unit 1131. the coded data Cgn from the encoder 1 110, the display cycle data Dp 
from the ON/OFF switch 1 141, and the output from the selector switch 11 42, to generate a multiplexed bit stream M1 . 
The multiplexer 1 120 outputs the multiplexed bit stream M1 as the coded image signal 100a having a fixed frame dis- 
play cycle or the coded image signal 100b having a variable frame display cycle. 
[01 1 0] The operation of the image coding apparatus 1 000 will be described briefly 

io [01 11] When an image signal Sg corresponding to a specific image is input to the apparatus 1000, the decision unit 
1131 decides whether the frame display cycle of the image signal Sg is variable or not, and outputs the display cycle 
identifier D1 indicating the result of the decision. Meanwhile, the first to third data generators 1 132-1 134 generate the 
display cycle data Dp, the frame number data Bn, and the display time data Dtn, respectively, based on the image signal 
Sg, and the encoder 1110 encodes the image signal Sg and outputs coded image data Cgn. 

is [01 1 2] The display cycle identifier Df and the coded image data Cgn are output toward the multiplexer 1 120. The dis- 
play cycle data Dp is transmitted through the ON/OFF switch 1 141, which is turned on or off according to the display 
cycle identifier Df, toward the multiplexer 1 1 20. The frame number data Bn and the display time data Dtn are transmitted 
through the selector switch 1 1 42, which selects one of the data Bn and the data Dtn according to the display cycle iden- 
tifier Df, toward the multiplexer 1 120. 

20 [01 1 3] That is, when an image signal having a fixed cycle of image display for each frame F(n) is input as the image 
signal Sg, the following data are output toward the multiplexer 1120: the fixed display cycle identifier Df, the display 
cycle data Dp indicating the cycle of image display of the respective frames, and the frame number data Bn correspond- 
ing to the respective frames and indicating the positional relations of the respective frames. In the multiplexer 1 120, the 
coded image data Cgn, the fixed display cycle identifier Df, the display cycle data Dp, and the frame number data Bn 

25 are multiplexed and output as the coded image signal 100a. 

[01 14] On the other hand, when an image signal having a variable cycle of image display of the respective frames 
F(n) is input as the image signal Sg, the following data are output toward the multiplexer 1 120: the variable display cycle 
identifier D1. and the display time data Dtn indicating the timing at which image display of each frame is carried out, i.e.. 
the display time h(n). which is set relatively to a desired one of plural reference times according to each frame. In the 

30 multiplexer 1 120. the coded image data Cgn, the variable display cycle identifier Df, and the display time data Dtn are 
multiplexed and output as the coded image signal 100b. 

[0115] Next, the process of decoding the coded image signals 100a and 100b will be described using figure 5. 
[0116] Initially, the display cycle identifier Df in the multiplexed bit stream M1 sent from the coding end (one of the 
coded image signals 100a and 100b) is detected to decide whether the display cycle of the coded image signal is fixed 
35 or not (step S21). When the result of the decision is that the display cycle is fixed, the count value n corresponding to 
the number of each frame F(n) in the transmission order is set to 0 (step S21a) and, thereafter, the display cycle data 
Dp indicating the display cycle T is read from the header of the coded image signal (step S22a). 
[01 17] Next, the frame number data Bn indicating the frame number B(n) is read from the header of each frame (step 
S23a), and the display time h(n) of each frame is calculated by h(n)=B(n)xT (step S24a). 
40 [0118] Then, the coded image data Cgn corresponding to the frame F(n) is decoded, and the decoded image data 
corresponding to the frame F(n) is regarded as data to be displayed at the display time h(n) (step S25a). Thereafter, it 
is decided whether or not the frame F(n) being processed (object frame) is the last frame of the specific image in the 
transmission order (step S26a). When the object frame is the last frame, the decoding process is completed. When the 
object frame is not the last frame, the count value n is incremented by 1 (step S27a) and, thereafter, the above- 
45 described steps S23a~S26a are repeated until it is decided in step S26a that the object frame is the last frame. 

[0119] On the other hand, when it is decided in step S21 that the display cycle is variable, the count value n, which 
corresponds to the number n of each frame F(n) in the transmission order, is set at 0 (step S21b) and, thereafter, the 
display time data Dtn indicating the display time h(n) of the frame F(n) is read from the header of each frame (step 
S22b), and the display time h(n) of the frame F(n) is decided according to the display time data Dtn (step S23b). Sub- 
so sequently, the coded image data Cgn corresponding to the frame F(n) is decoded, and the decoded image data of the 
frame F(n) is regarded as data to be displayed at the display time h(n) (step S24b). 

[0120] Thereafter, it is decided whether or not the frame F(n) being processed (object frame) is the last frame of the 
specific image in the transmission order (step S25b). When the object frame is the last frame, the decoding process is 
completed. When the object frame is not the last frame, the count value n is incremented by 1 (step S26b). Thereafter. 
55 the above-described steps S23a-S26a are repeated until it is decided in step S25b that the object frame is the last 
frame. 

[0121] As described above, the coded image signals having the data structures shown in figures 1(a) and 1(b) are 
decoded in the procedure shown in figure 5. 
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[0122] Figure 6(a) is a block diagram illustrating the structure of an image decoding apparatus as hardware perform- 
ing the decoding process according to the first embodiment. 

[0123] The image decoding apparatus 2000 receives the multiplexed bit stream M1 (the coded image signal 100a or 
100b) output from the image coding apparatus 1000, and subjects the multiplexed bit stream M1 to reproduction includ- 

5 ing decoding and display. 

[0124] More specifically, the image decoding apparatus 2000 includes a demultiplexer (DEMUX) 21 10 for extracting, 
from the multiplexed bit stream M1 , the coded image data Cgn, the display cycle identifier Df, the display cycle data Dp, 
and the frame number data Bn or the display time data Dtn, and outputting these data; and a decoder 21 20 for decoding 
the coded image data Cgn and outputting decoded image data Rg. 

io [0125] Further, the image decoding apparatus 2000 includes an ON/OFF switch 2140 for switching the circuit between 
the ON state where the display cycle data Dp is transmitted and the OFF state where the display cycle data Dp is cut 
off; and a selector switch 2150 for selecting one of the frame number data Bn and the display time data Dtn, based on 
the display cycle identifier Df, and outputting the selected data. 

[0126] Furthermore, the image decoding apparatus 2000 includes a display unit 2130 for displaying the decoded 
15 image data Rg at a prescribed timing based on the display cycle identifier Df and the outputs from the switches 2140 
and 2150. 

[0127] Hereinafter, the operation of the image decoding apparatus 2000 will be briefly described. 
[0128] When the multiplexed bit stream Ml from the image coding apparatus 1000 is input to the image decoding 
apparatus 2000, in the demultiplexer 21 10, the display cycle identifier Df and the display cycle data Dp are separated 
20 from the multiplexed bit stream M1 . Further, the coded image data Cgn and the frame number data Bn or the display 
time data Dtn are separated for each frame from the multiplexed bit stream M1 . 

[0129] The coded image data Cgn of each frame is decoded by the decoder 2120 and then output as decoded image 
" data Rg toward the display unit 2130. The display cycle data Dp is transmitted through the ON/OFF switch 2140 which 
is turned on or off by the display cycle identifier Df toward the display unit 2130, and the frame number data Bn or the 
25 display time data Dtn of each frame is transmitted through the selector switch 2150, which selects one of these data 
according to the display cycle identifier Df. toward the display unit 2130. 

[0130] In the display unit 2130, the image of each frame corresponding to the decoded image data Rg having a fixed 
display cycle is displayed at a prescribed display timing based on the display cycle data Dp and the frame number data 
Bn, while the image of each frame corresponding to the decoded image data Rg having a variable display cycle is dis- 

30 played at a prescribed display timing based on the display time data Dtn. 

[0131] As described above, in the image signal data structure according to the first embodiment, since the coded 
image signal obtained by coding an image signal includes the display cycle identifier Df indicating whether the cycle of 
image display for each frame is variable or not, when the cycle of image display for each frame is fixed, the decoded 
image data Rg can be displayed by a simple circuit structure, i.e., based on the display cycle data Dp and the frame 

35 number data Bn having a relatively small data quantity (bit number), without referring to the display time data (display 
timing data) Dtn having a relatively large data quantity (bit number) for each frame. 

[0132] Further, since the coded image signal 100a having a fixed display cycle includes the display cycle data Dp indi- 
cating the image display cycle T and the data (frame position data) Bn indicating the frame number B(n) which indicates 
the positional relationship of the frame with the previous and subsequent frames, the image display timing of each 

40 frame can be decided by a simple calculation, TxB(n) . 

[0133] Furthermore, the coded image signal 100b having a variable display cycle includes the display time data (dis- 
play timing data) Dtn indicating the display time (display timing) h(n) at which the image of each frame is to be displayed, 
which display time is set relatively to a desired time h (0) among plural reference times, according to each frame (refer 
to figure 2). Therefore, when the cycle of image display of each frame is variable, like the conventional data structure, 

45 the image display timing h(n) of each frame F(n) can be set based on the display time data Dtn. 

[0134] Furthermore, the image coding apparatus 1000 according to this first embodiment includes the decision unit 
1131 for generating the display cycle identifier Df which indicates whether the image display cycle is variable or fixed, 
based on the input image signal. When the input image signal has a fixed display cycle, the coded image data Cgn is 
multiplexed with the display cycle identifier Df, the display cycle data Dp indicating the cycle of image display, and the 

so frame number data Bn indicating the positional relationship of each frame with the previous and subsequent frames, 
and the data so multiplexed are output. When the input image signal has a variable display cycle, the coded image data 
Cgn is multiplexed with the display cycle identifier Df and the display time data Dtn indicating the display time h(n) of 
each frame F(n), and the data so multiplexed are output. Therefore, whether the cycle of image display for each frame 
is variable or fixed, the data for deciding the display timing of each frame is output together with the coded image data 

55 Cgn. Therefore, the number of bits required for deciding the display time can be reduced when the frame display cycle 
is fixed. Further, the image having a variable frame display cycle can be displayed in the same manner as the conven- 
tional display. 

[01 35] The image decoding apparatus 2000 according to this first embodiment includes the demultiplexer 2110 which 
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separates the display cycle identifier Df, the display cycle data Dp indicating the cycle of image display, the frame 
number data Bn indicating the positional relationship of each frame, the display time data Dtn indicating the display time 
h(n) of each frame, and the coded image data Cgn, from the multiplexed bit stream M1 transmitted from the image cod- 
ing apparatus 1 000. Further, the apparatus 2000 includes the decoder 21 20 which decodes the coded image data Cgn 

5 and outputs the decoded image data Rg. In this apparatus, the decoded image data Rg having a fixed display cycle is 
displayed at a prescribed display timing based on the display cycle data Dp and the frame number data Bn, while the 
decoded image data Rg having a variable display cycle is displayed at a prescribed display timing based on the display 
time data Dtn. Therefore, whether the frame display cycle of the decoded image data Rg is variable or fixed, the 
decoded image data Rg of each frame can be displayed at an accurate display timing. 

io [0136] In the image data structure according to the first embodiment, the display cycle identifier Df is inserted at the 
beginning of the image data (multiplexed bit stream), and the frame number data Bn or the display time data Dtn is 
inserted at the beginning of each frame data (code sequence of each frame). However, the display cycle identifier, the 
frame number data, and the display time data are not necessarily inserted at the beginnings of the corresponding head- 
ers. These data may be inserted after synchronous signals or the like as along as the display cycle identifier and the 

is display cycle data are included in the header of the image data (coded image signal) while the frame number data and 
the display time data are included in the header of the data (code sequence) corresponding to the frame. 
[01 37] While in this first embodiment the display cycle data Dp is inserted just after and next to the display cycle iden- 
tifier Df, the display cycle data Dp may be inserted in any place after the display cycle identifier Df within the header of 
the image data. 

20 [01 38] Further, while in this first embodiment serial numbers are given, as the numbers n'(=B(n)) indicating the display 
order, from the beginning of the image data in the display order as shown in figure 2, it is not always necessary to give 
such serial numbers. A plurality of numbers from the first number to the last number, which have been determined in 
* ' - advance, may be given periodically as the frame numbers. 

[0139] For example, when the frame numbers are expressed with 4 bits, numbers from 0 to 15 are periodically given 

25 to the frames. In this case, the display time is expressed by 

h , (n')=hp'(15)+(n , +1)xT 

where hp'(n') indicates the display time corresponding to the frame number B(n) (=n') in the previous period. Accord- 
30 ingly, h'(n') indicates the display time corresponding to the frame number B(n) (=n') in the cycle next to hp'(n'). In the 
formula, hp'(15) indicates the display time of the last frame in the previous period. 

[0140] Furthermore, while in this first embodiment each frame is specified by using the frame number data, any data 
may be used as long as it specifies the positional relationship of each frame with the previous and subsequent frames, 
for example, data indicating the positional relationships of the respective frames according to a specific rule or data 

35 specifying the positional relationships of the respective frames with reference to a prescribed table. 

[0141] Moreover, in this first embodiment, the display time data indicates the time relative to plural reference times, 
and one reference time may be set for plural frames or the display time of the previous frame may be used as the ref- 
erence time. Further, plural reference times may be set in advance, and which reference time is to be used for express- 
ing the frame display time may be decided according to some rule or signal. 

40 [0142] Further, the data structure of the coded image signal according to this first embodiment includes the display 
cycle identifier, the display cycle data, and the frame number data or the display time data, these data being used for 
setting the display timing of each frame, as the additional data for deciding the reproduction timing of each frame at the 
decoding end. However, instead of those additional data for deciding the display timing of each frame, the data structure 
of the coded image signal may includes additional data for deciding the decoding timing of each frame, namely, decod- 

45 ing cycle identifier, decoding cycle data, and frame number data or decoding time data. Hereinafter, such data structure 
will be described as a modification of the first embodiment. 

(Modification of Embodiment 1 ) 

so [0143] In the data structure according to the modification of the first embodiment, the display cycle identifier Df and 
the display cycle data Dp in the coded image signal 100a of the first embodiment are replaced with a decoding cycle 
identifier and decoding cycle data, and the display time data Dtn in the coded image signal 100b of the first embodiment 
is replaced with decoding time data. 

[0144] The decoding cycle identifier indicates whether the cycle of a decoding process for decoding a coded image 
55 signal of each frame is variable or not. and a fixed decoding cycle identifier is inserted in a coded image signal having 
a fixed decoding cycle while a variable decoding cycle identifier is inserted in a coded image signal having a variable 
decoding cycle. 

[0145] Further, the decoding cycle data indicates the cycle DT of decoding each frame, and the decoding time data 
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indicates the timing at which decoding of each frame is carried out (decoding time Dh(n)), which timing is set relatively 
to a desired one of plural reference times according to each frame. 

[0146] The coding process for generating the coded image signal having the data structure according to this modifi- 
cation of the first embodiment is realized by replacing steps S11. Sl1a, S11b, S13a and S13b in the flow of figure 3 as 

5 follOWS. 

[0147] That is, the process of deciding the display cycle in step S1 1 is replaced with deciding whether the decoding 
cycle is fixed or not; the process of adding the fixed display cycle identifier Df in step S1 1a is replaced with the process 
of adding the fixed decoding cycle identifier; and the process of adding the variable display cycle identifier Of in step 
Sl1b is replaced with the process of adding the variable decoding cycle identifier. Further, the process of adding the 
io display cycle data Dp in step S13a is replaced with the process of adding the decoding cycle data; and the process of 
adding the display time data Dtn in step Si 3b is replaced with the process of adding the decoding time data. 
[0148] Figure 4(b) shows the structure of an image coding apparatus 1000a as hardware for performing the coding 
process according to the modification of the first embodiment. 

[0149] The image coding apparatus 1000a includes a decision unit 1 131a for deciding whether the frame decoding 
is cycle of the input image signal Sg is fixed or not, i.e., fixed or variable, and outputting a decoding cycle identifier Off indi- 
cating whether the decoding cycle DT is fixed or not, in place of the decision unit 1 131 of the image coding apparatus 
1000 of the first embodiment. 

[0150] Further, the image coding apparatus 1000a includes a decoding cycle data generator (first data generator) 
1132a which generates decoding cycle data DEp indicating the frame decoding cycle (fixed cycle) DT, based on the 

20 input image signal Sg. and a decoding time data generator (third data generator) 1 1 34a which generates decoding time 
data DEtn indicating the decoding time of each frame, based on the input image signal Sg, in place of the display cycle 
data generator 1 132 and the display time generator 1 134 of the image coding apparatus 1000. 
[0151] Other constituents are identical to those of the image coding apparatus 1000 of the first embodiment. 
[0152] In the image coding apparatus 1000a so constructed, the multiplexer (MUX) 1 102 multiplexes the decoding 

25 cycle identifier DEf from the decision unit 1131a, the coded image data Cgn from the encoder 1110, the decoding cycle 
data DEp from the ON/OFF switch 1141, and the output from the selector switch 1142, and outputs a multiplexed bit 
stream M 1 a as a coded image signal having a fixed decoding cycle or a coded image signal having a variable decoding 
cycle. 

[0153] On the other hand, the process for decoding the coded image signal having the data structure according to 
30 this modification of the first embodiment is realized by replacing steps S21 , S22a. S22b, S23a, S23b, S24a, S24b and 
S25a in the flow of figure 5 as follows. 

[0154] That is, the process of deciding the display cycle in step S21 is replaced with the process of deciding whether 
the decoding cycle is fixed or not; the process of reading the display cycle data Dp indicating the display cycle T in step 
S22a is replaced with the process of reading the decoding cycle data DEp indicating te decoding cycle Dt; and the proc- 
35 ess of reading the display time data Dtn indicating the display time h(n) in step S22b is replaced with the process of 
reading the decoding time data DEtn indicating the decoding time Dh(n). 

[01 55] Further, the process of deciding the display time h(n) by reading the frame number data Bn in steps S23a and 
S24a is replaced with the process of deciding the decoding time Dh(n) of coded image data of each frame based on the 
decoding cycle data DEp and then deciding the display time h(n) of each frame based on the frame number data Bn. 
40 [0156] Further, the process of deciding the display time h(n) based on the display time data Dtn in step S23b is 
replaced with the process of deciding the decoding time Dh(n) based on the decoding time data DEtn and then deciding 
the display time h(n) based on the data DEtn. 

[01 57] Furthermore, the process of decoding the coded image data Cgn of the frame F(n) and displaying it at the dis- 
play time h(n) in step S25a is replaced with the process of decoding the coded image data Cgn of the frame F(n) at the 

45 decoding time Dh(n) and displaying it at the display time h(n); and the process of decoding the coded image data Cgn 
of the frame F(n) and displaying it at the time h(n) in step S24b is replaced with the process of decoding the coded 
image data Cgn of the frame F(n) at the decoding time Dh(n) and displaying it at the display time h(n). 
[0158] Figure 6(b) is a block diagram illustrating the structure of an image decoding apparatus 2000a as hardware 
performing the decoding process according to the modification of the first embodiment. 

so [0159] The image decoding apparatus 2000a receives the multiplexed bit stream M1a output from the image coding 
apparatus 1000a, and subjects the multiplexed bit stream M1a to reproduction including decoding and display. 
[0160] More specifically, the image decoding apparatus 2000a includes a demultiplexer (DEMUX) 21 10a for extract- 
ing, from the multiplexed bit stream M1a. the coded image data Cgn, the decoding cycle identifier DEf, the decoding 
cycle data DEp. and the frame number data Bn or the decoding time data DEtn, and outputting these data, in place of 

55 the demultiplexer 21 10 of the image decoding apparatus 2000 of the first embodiment. 

[0161] Further, the image decoding apparatus 2000a includes a first ON/OFF switch 2140a for switching the circuit 
between the ON state where the decoding cycle data DEp is transmitted and the OFF state where the decoding cycle 
data DEp is cut off, based on the decoding cycle identifier DEf; a second ON/OFF switch 2150a for switching the circuit 
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between the ON state where the frame number data Bn is transmitted and the OFF state where the frame number data 
Bn is cut off, based on the decoding cycle identifier DEf; and a third ON/OFF switch 2160a for switching the circuit 
between the ON state where the decoding time data DEtn is transmitted and the OFF state where the decoding time 
data DEtn is cut off, based on the decoding cycle identifier DEf. 
5 [0162] In the image decoding apparatus 2000a, the decoding cycle data DEp output from the first ON/OFF switch 
2140a and the decoding time data DEtn output from the third ON/OFF switch 2160a are supplied to the decoder 21 20a 
and the display unit 2130a, while the frame number data Bn output from the second ON/OFF switch 2150a is supplied 
to the display unit 2130a. 

[0163] In the decoder 2120a, the coded image data Cgn having a fixed decoding cycle is decoded frame by frame at 
w the timing (decoding time Dh(n)) decided by the decoding cycle data DEp, while the coded image data Cgn having a 
variable decoding cycle is decoded frame by frame at the timing (decoding time Dh(n)) decided by the decoding time 
data DEtn. 

[0164] Further, in the display unit 2130a, the decoded image data Rg having a fixed decoding cycle is displayed frame 
by frame at the timing (display time h(n)) decided by the decoding cycle data DEp and the frame number data Bn, while 
is the decoded image data Rg having a variable decoding cycle is displayed frame by frame at the timing (display time 
h(n)) decided by the decoding time data DEtn. 

[0165] Other constituents of the image decoding apparatus 2000a are identical to those of the image decoding appa- 
ratus 2000 of the first embodiment. 

[0166] Hereinafter, the operation of the image decoding apparatus 2000a will be described briefly. 
20 [0167] When the multiplexed bit stream M1a is input to the apparatus 2000a, in the demultiplexer 21 10a, the coded 
image data Cgn, the decoding cycle identifier DEf, the decoding cycle data DEp, and the frame number data Bn or the 
decoding time data DEtn are separated from the input bit stream M1a. 

[01 68] In the decoder 2 1 20a, when the decoding cycle of the input coded image signal is fixed, the coded image data 
Cgn is decoded frame by frame at the timing (decoding time Dh(n)) decided by the decoding cycle data DEp. Then, the 
25 decoded image data Rg output from the decoder 2120a is displayed frame by frame at the timing (display time h(n)) 
decided by the decoding cycle data DEp and the frame number data Bn. 

[01 69] On the other hand, when the decoding cycle of the input coded image signal is variable, the coded image data 
Cgn is decoded frame by frame at the timing (decoding time Dh(n)) decided by the decoding time data DEtn. Then, the 
decoded image data Rg output from the decoder 2120a is displayed frame by frame at the timing (display time h(n)) 

30 decided by the decoding time data DEtn. 

[0170] In the modification of the first embodiment, as in the first embodiment, since the coded image signal obtained 
by coding an image signal includes the decoding cycle identifier DEf indicating whether the cycle of image decoding for 
each frame is variable or not, when the cycle of image decoding for each frame is fixed, the coded image data can be 
decoded by a simple circuit structure, that is, it can be decoded according to the decoding cycle data DEp alone, without 

35 referring to the decoding time data DEtn having a large data quantity (bit number) for each frame. 

[01 71 ] In the modification of the first embodiment, emphasis has been placed on an image decoding apparatus which 
performs image display of each frame as well as decoding of each frame according to data for deciding the decoding 
timing of each frame, which data is included in the coded image signal. However, an image decoding apparatus which 
performs decoding of each frame as well as image display of each frame according to data for deciding the display tim- 

40 ing of each frame, which data is included in a coded image signal, is also within the scope of the present invention. 
[0172] In this case, the decoding timing at which decoding of each frame is carried out is set according to display tim- 
ing data of plural frames including an object frame to be decoded. That is, based on the display timing data of the object 
frame and the display timing data of the next frame transmitted subsequently to the object frame, the decoding timing 
of the object frame is set at a timing that is earlier, by a prescribed offset time, than the earlier display timing between 

45 the display timings of the object frame and the next frame. 

[0173] To be specific, when the display timing of the object frame is earlier than the display timing of the next frame 
transmitted subsequently to the object frame, the offset time is set to a length longer than the time required for decoding 
the object frame. On the other hand, when the display timing of the next frame (for example, B-VOP) transmitted sub- 
sequently to the object frame (for example, P-VOP) is earlier than the display timing of the object frame, the offset time 

so is set to a length longer than the sum of the time required for decoding the object frame and the time required for decod- 
ing the next frame. 

[Embodiment 2] 

55 [0174] Figure 7(a) shows a data structure of a coded image signal 120a having a fixed frame display cycle, according 
to a second embodiment of the present invention. 

[0175] The coded image signal 120a is obtained by coding an image signal which corresponds to one image (in 
MPEG4, an image corresponding to one object) and has a fixed frame display cycle. The coded image signal 120a has 
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a header H at the beginning, and the header H is followed by code sequences ScO, Scl . Sc2 Sen corresponding to 

frames F(0). F(1), F(2) t .... F(n), respectively, which code sequences are arranged in the transmission order. In the 
coded image signal 120a r the following data are included in the header H: a display cycle identifier Df indicating whether 
the frame display cycle is fixed or not; display cycle multiplier data Dm indicating that the frame display cycle is the sub- 

5 unit time (1/N) multiplied by M (natural number) by the multiplier M, i.e., how many times (M) the frame display cycle is 
as long as the sub-unit time (1/N); and sub-unit time data Dk indicating the value N (natural number) for obtaining the 

sub-unit time (1/N). Further, at the beginnings of the code sequences ScO, Scl , Sc2 Sen of the respective frames, 

display time data DyO, Dyl , Dy2, .... Dyn indicating the display times y'O, y'3. y'1 y'n* of the respective frames (refer 

to figure 17(a)) are inserted. In the header H of the coded image signal 120a, the sub-unit time data Dk, the display 

w cycle identifier Df, and the display cycle multiplier data Dm are arranged so that these data are transmitted in this order. 
[0176] In the code sequences ScO, Scl, Sc2, .... Sen of the respective frames, the display time data DyO, Dyl, Dy2, 
.... Dyn are followed by coded image data CgO, Cgl , Cg2 Cgn, respectively. 

[0177] In this coded image signal 120a, assuming that the reference time is x (refer to figure 17(a)), the display times 
h(0), h(1), h(2), ... of the frames F(0), F(1), F(2), ... corresponding to the VOP0, VOP3, VOP1, ... are expressed by 

15 x+y/N(y=y"0, y'3. y'1 . ...) based on the display time data DyO, Dy1 , Dy2 ... 

[0178] However, since the coded image signal 120a includes the sub-unit time data Dk and the display cycle multiplier 
data Dm. image display of each frame can be carried out without using the display time data (DyO, Dy1 , Dy2, ...). To be 
specific, based on the sub-unit time (1/N) obtained from the sub-unit time data Dk and the value of M (natural number) 
obtained from the display cycle multiplier data Dm, the frame display cycle T (=Mx1/N) is obtained, and the image of 

20 each frame is displayed at the original display time h(n) (=x+yxMx1/N) of each frame F(n), decided by the reference 
time x. 

[0179] Figure 7(b) illustrates the data structure of a coded image signal 120b having a variable frame display cycle, 
- according to the second embodiment of the invention. 
[0180] The coded image signal 120b is different from the coded image signal 120a in that the header H does not 
25 include the display cycle multiplier data Dm. 

[0181] Hereinafter, a description is given of a coding process for generating the coded image signal 120a or 120b and 
a decoding process for decoding the coded image signal. 

[0182] Figure 8 is a flowchart of the coding process. In the coding process, initially, the sub-unit time data Dk is added 
to the header of the bit stream of the input image signal corresponding to a specific image (step S30), and it is decided 

30 whether the display cycle of the image signal is fixed or not (step S3 1 ). When the result of the decision is that the display 
cycle is fixed, a fixed display cycle identifier Df indicating that the display cycle of the image signal is fixed is added to 
the header of the bit stream so that it follows the sub-unit time data Dk (step S32) and, furthermore, the display cycle 
multiplier data Dm is added to the header so that it follows the fixed display cycle identifier of (step S33). 
[0183] Thereafter, the count value n, which corresponds to the number n indicating the transmission order of each 

35 frame F(n) as a constituent of the specific image, is set to 0 (step S35). 

[0184] Next, as a code sequence SaO corresponding to the first frame F(0) of the specif ic image in the transmission 
order, display time data Dyn (=Dy0) and coded image data Cgn (=Cg0) of the frame are successively added to the 
header H (steps S36 and S3 7). Thereafter, it is decided whether or not the frame being processed (hereinafter, referred 
to as "object frame") in the image signal is the last frame of the specific image in the transmission order (step S38). 

40 When the object frame is not the last frame, the count value n corresponding to the frame F(n) (=F(0)). the transmission 
order of which is n-th, is incremented by 1 (step S39), and the following frame F(n+1) (=F(1)) is subjected to the proc- 
esses of steps S36-S39. 

[0185] The processes of steps S36-S39 are repeated until it is decided in step S38 that the object frame is the last 
frame. The coded image signal 120a is generated through the above-described process steps. 
45 [0186] On the other hand, when the result of the decision in step S31 is that the display cycle is variable, a variable 
display cycle identifier Df indicating that the display cycle of the image signal is variable is added to the header H of the 
bit stream corresponding to the image signal so that it follows the sub-unit time data Dk (step S34). Thereafter, the proc- 
esses of steps S35-S39 are carried out, thereby generating the coded image signal 120b. 

[0187] Figure 9(a) is a block diagram illustrating an image coding apparatus 1200 as hardware performing the coding 
so process according to the second embodiment. 

[0188] The image coding apparatus 1200 includes an encoder 1 1 10 for coding an input image signal Sg to generate 
coded image data Cgn, and a decision unit 1 131 for deciding whether the frame display cycle of the input image signal 
Sg is fixed or not, i.e., fixed or variable, and outputting a display cycle identifier Df indicating whether the display cycle 
is fixed or not. 

55 [0189] Further, the image coding apparatus 1200 includes three data generators as follows: a sub-unit time data gen- 
erator (first data generator) 1232 for generating sub-unit time data Dk based on the input image signal Sg; a display 
cycle multiplier data generator (second data generator) 1233 for generating display cycle multiplier data Dm which indi- 
cates a numeric value M for expressing the frame display cycle by a multiplier (M) for the sub-unit time, based on the 
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input image signal Sg; and a display time data generator (third data generator) 1234 for generating display time data 
(display timing data) Dyn indicating the display time h(n) of each frame, based on the input image signal Sg. 
[0190] Furthermore, the image coding apparatus 1200 includes an ON/OFF switch 1241 for switching the circuit 
between the ON state where the display cycle multiplier data Dm is transmitted and the OFF state where the display 

5 cycle multiplier data Dm is cut off, based on the display cycle identifier Df from the decision unit 1131. 

[0191] The image coding apparatus 1200 further includes a multiplexer (MUX) 1220 for multiplexing the sub-unit time 
data Dk from the first data generator 1232, the display cycle identifier of from the decision unit 1131, the display cycle 
multiplier data Dm from the switch 1241, the display time data Dyn from the third data generator 1234, and the coded 
image data Cgn from the encoder 1 1 1 0, to generate a multiplexed bit stream M2. The multiplexer 1 220 outputs the mul- 

io tiplexed bit stream M2 as the coded image signal 120a or the coded image signal 120b. 
[0192] The operation of the image coding apparatus 1200 will be described briefly 

[0193] When an image signal Sg corresponding to a specific image is input to the apparatus 1200, the decision unit 
1131 decides whether the display cycle of the image signal Sg is variable or not, and outputs the display cycle identifier 
Df indicating the result of the decision. Meanwhile, the first to third data generators 1232-1234 generate the sub-unit 
75 time data Dk, the display cycle multiplier data Dm, and the display time data Dyn, respectively, based on the image sig- 
nal Sg, and the encoder 1 1 10 encodes the image signal Sg and outputs coded image data Cgn. 

[0194] The sub-unit time data Dk, the display cycle identifier Df, the display time data Dyn, and the coded image data 
Cgn are continually output toward the multiplexer 1220. The display cycle multiplier data Dm is transmitted through the 
switch 1241 , which is in the ON state by the display cycle identifier Df, toward the multiplexer 1220. 

20 [0195] That is, when an image signal having a fixed cycle of image display is input as the image signal Sg, the sub- 
unit time data Dk, the display cycle identifier Df, the display cycle multiplier data Dm, the display time data Dyn of each 
frame, and the coded image data Cgn of each frame are output to the multiplexer 1220. In the multiplexer 1220, the sub- 
m ~" unit time data Dk, the display cycle identifier Df, the display cycle multiplier data Dm, the coded image data Cgn, and 
the display time data Dyn are multiplexed, and the coded image signal 120a is output as the multiplexed bit stream M2. 

25 [0196] On the other hand, when an image signal having a variable cycle of image display is input as the image signal 
Sg. the switch 1241 goes into the OFF state according to the display cycle identifier Df. In this state, the sub-unit time 
data Df, the display cycle identifier Df , the display time data Dyn of each frame, and the coded image data Cgn of each 
frame are output to the multiplexer 1220. In the multiplexer 1220, the sub-unit time data Df, the display cycle identifier 
Df, the display time data Dyn, and the coded image data Cgn are multiplexed, and the coded image signal 120b is out- 

30 put as the multiplexed bit stream M2. 

[0197] Next, a description is given of a decoding process for decoding a coded image signal having a data structure 
according to this second embodiment, with reference to figure 10. 

[0198] Figure 10 is a flowchart of the decoding process. In the decoding process, initially, the sub-unit time data Dk 
included in the multiplexed bit stream M2 (coded image signal 120a or 120b) sent from the coding end is read (step 

35 S40), and the display cycle identifier Df is detected to decide whether the display cycle of the coded image signal is 
fixed or not (step S41). When it is decided that the display cycle is fixed, the display cycle multiplier data Dm, which 
expresses the display cycle T by a multiplier M for the sub-unit time (1/N), is read from the header H of the coded image 
signal (step S42a). Subsequently, based on the sub-unit time data Dk and the display cycle multiplier data Dm, the 
frame display cycle T is obtained by T = (VN)xM (step S43a). Thereafter, the count value n\ which corresponds to the 

40 number n* of each frame F'(n') from the first frame in the display order, is set to 0 (step S44a), and the display time h(n') 
of each frame F'(n') is obtained by h'(n') = n'xT (step S45a). At this time, the coded image data Cgn corresponding to 
each frame F(n) is decoded in the transmission order, generating decoded image data Rg corresponding to the frame 
F(n). 

[0199] Thereafter, it is decided whether or not the object frame F'(n') counted in the display order is the last frame of 
45 the specific image (step S46a). When the object frame is the last frame, the decoding process is completed. When the 
object frame is not the last frame, the count value n' is incremented by 1 (step S47a), and the processes of steps 
S45a~S47a are repeated until it is decided in step S46a that the object frame is the last frame. 

[0200] In the decoding process, the decoded image data Rg corresponding to each decoded frame F(n') is displayed 
at the display time h'(n') in the prescribed display order n\ 

so [0201] On the other hand, when it is decided that the display cycle is variable in step S41 , the count value n corre- 
sponding to the number n of each frame F(n) in the transmission order is set to 0 (step S42b). Subsequently, the display 
time data Dyn indicating the display time h(n) of each frame F(n) is read from the header H of the frame F(n) (step 
S43b). and the display time h(n) of each frame F(n) is obtained according to the display time data Dyn (step S44b). At 
this time, the coded image data Cgn of each frame F(n) is decoded in the transmission order. 

55 [0202] Thereafter, it is decided whether or not the object frame F(n) counted in the transmission order is the last frame 
of the specific image (step S44b). When the object frame is the last frame, the decoding process is ended. When the 
object frame is not the last frame, the count value n is incremented by 1 (step S46b) and, thereafter, the processes of 
steps S42b-S46b are repeated until it is decided in step S45b that the object frame is the last frame. 
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[0203] In the decoding process, the decoded image data Rg corresponding to each decoded frame F(n) is displayed 
at the display time h(n) of the frame F(n) in the prescribed display order n\ 

[0204] Figure 11(a) is a block diagram illustrating the structure of an image decoding apparatus 2200 as hardware 
performing the decoding process of the second embodiment. 
5 [0205] The image decoding apparatus 2200 decodes and reproduces the multiplexed bit stream M2 (the coded image 
signal 120a or 120b) output from the image coding apparatus 2000. 

[0206] More specifically, the image decoding apparatus 2200 includes a demultiplexer (DEMUX) 2210 for extracting, 
from the multiplexed bit stream M2, the sub-unit time data Dk, the display cycle identifier Df, the display cycle multiplier 
data Dm, the display time data Dyn, and the coded image data Cgn, and outputting these data; and a decoder 2220 for 

10 decoding the coded image data Cgn and outputting decoded image data Rg. 

[0207] Further, the image decoding apparatus 2200 includes a first ON/OFF switch 2240 for switching the circuit 
between the ON state where the display cycle multiplier data Dm is transmitted and the OFF state where the display 
cycle multiplier data Dm is cut off, according to the display cycle identifier Df ; and a second ON/OFF switch 2250 for 
switching the circuit between the ON state where the display time data Dyn is transmitted and the OFF state where the 

75 display time data Dyn is cut off, according to the display cycle identifier Df. 

[0208] Furthermore, the image decoding apparatus 2200 includes a display unit 2230 which receives the display cycle 
multiplier data Dm and the display time data Dty through the first and second ON/OFF switches 2240 and 2250, respec- 
tively, as well as the sub-unit time data Dk and the decoded image data Rg, and performs image display at the pre- 
scribed display timing based on these data. 

20 [0209] The operation of the image decoding apparatus 2200 will be described briefly. When the multiplexed bit stream 
M2 from the image coding apparatus 1200 is input to the image decoding apparatus 2200, in the demultiplexer 2210, 
the sub-unit time data Dk, the display cycle identifier Df, and the display cycle multiplier data Dm are separated from the 
"multiplexed bit stream M2 and, furthermore, the display time data Dyn and the coded image data Cgn are separated 
frame by frame from the bit stream M2. 

25 [021 0] The coded image data Cgn of each frame is decoded by the decoder 2220 and then output as decoded image 
data Rg toward the display unit 2230. Meanwhile, the sub-unit time data Dk is output directly to the display unit 2230. 
The display cycle multiplier data Dm is output through the first ON/OFF switch 2240, which is turned on or off by the 
display cycle identifier Df, toward the display unit 2230, while the frame display time data Dyn is output through the sec- 
ond ON/OFF switch 2250, which is turned on or off by the display cycle identifier Df, toward the display unit 2230. When 

30 the multiplexed bit stream M2 is the coded image signal 1 20a having a fixed display cycle, the first and second ON/OFF 
switches 2240 and 2250 are in the ON states, and when the multiplexed bit stream M2 is the coded image signal 120b 
having a variable display cycle, the first and second ON/OFF switches 2240 and 2250 are in the OFF states. 
[021 1] In the display unit 2230, the image of each frame corresponding to the decoded image data Rg having a fixed 
display cycle is displayed at a prescribed display timing based on the sub-unit time data Dk and the display cycle mul- 

35 tiplier data Dm. In this case, the display timing of each frame is the display time h'(n') which is obtained by an arithmetic 
expression, h'(n') = Txn' (T=(1/N)xM) . On the other hand, the image of each frame corresponding to the decoded 
image data Rg having a variable display cycle is displayed at a prescribed timing based on the display time data Dty. In 
this case, the display timing is the display time h(n) which is decided by the display time data Dty. 
[0212] As described above, according to the second embodiment of the invention, since the data structure of the 

40 coded image signal includes the sub-unit time data Dk indicating the length of the sub-unit time (1/N), which is obtained 
by dividing a prescribed time interval with N (natural number), by the natural number N, and the display cycle multiplier 
data Dm indicating the fixed frame display cycle T by a multiplier M for the sub-unit time (1/N), in addition to the display 
cycle identifier Df indicating whether the display cycle of each frame is variable or not. Therefore, when processing a 
coded image signal having a fixed frame rate, the value of the fixed frame rate of the coded image signal can be 

45 detected before decoding each frame, whereby the various hardware structures for implementing the image display can 
be simplified. 

[021 3] Further, the data structure of the coded image signal according to the second embodiment includes the sub- 
unit time data Dk, the display cycle identifier Df, the display cycle multiplier data Dm, and the display time data Dyn. 
these data being used for setting the display timing of each frame, as additional data for deciding the timing of repro- 
50 duction for each frame at the decoding end. However, in place of those additional data for deciding the display timing of 
each frame, the data structure of the coded image signal may includes additional data for deciding the timing of decod- 
ing each frame, i.e., sub-unit time data, decoding cycle identifier, decoding cycle multiplier data, and decoding time 
data. Hereinafter, such data structure will be described as a modification of the second embodiment. 

55 (Modification of Embodiment 2) 

[0214] In the data structure according to the modification of the second embodiment, the display cycle identifier Df 
and the display cycle multiplier data Dm in the coded image signal 120a of the second embodiment are replaced with 
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a decoding cycle identifier DEf and decoding cycle multiplier data DEp, and the display time data Dyn in the coded 
image signal 120b of the second embodiment is replaced with decoding time data DEyn. 

[0215] The decoding cycle identifier DEf indicates whether the cycle of a decoding process for decoding a coded 
image signal of each frame is variable or not. A fixed decoding cycle identifier DEf is inserted in a coded image signal, 
5 the decoding cycle DT of which is fixed, while a variable decoding cycle identifier DEf is inserted in a coded image sig- 
nal, the decoding cycle DT of which is variable. 

[0216] The decoding cycle multiplier data DEm indicates the frame decoding cycle DT by a multiplier M for the sub- 
unit time (1/N), i.e., it indicates how many times (M) the decoding cycle DT is as long as the sub-unit time. The decoding 
time data DEyn indicates the timing at which decoding of each frame is carried out. 
10 [0217] The coding process for generating the coded image signal having the data structure according to this modifi- 
cation of the second embodiment is realized by replacing steps S31 , S32, S33, S34, and S36 in the flow of figure 8 as 
follows. 

[0218] That is, the process of deciding the display cycle in step S31 is replaced with the process of deciding whether 
the decoding cycle is fixed or not; the process of adding the fixed display cycle identifier Df in step S32 is replaced with 
is the process of adding the fixed decoding cycle identifier DEf; and the process of adding the variable display cycle iden- 
tifier Df in step S34 is replaced with the process of adding the variable decoding cycle identifier DEf. Further, the proc- 
ess of adding the display cycle multiplier data Dm in step S33 is replaced with the process of adding the decoding cycle 
multiplier data DEm; and the process of adding the display time data Dyn in step S36 is replaced with adding the decod- 
ing time data DEyn. 

20 [0219] Figure 9(b) shows the structure of an image coding apparatus 1200a as hardware for performing the coding 
process according to the modification of the second embodiment. 

[0220] The image coding apparatus 1 200a includes a decision unit 1 1 31 a for deciding whether the frame decoding 
cycle of the input image signal Sg is fixed or not, i.e., fixed or variable, and outputting a decoding cycle identifier DEf 
indicating whether the decoding cycle is fixed or not. in place of the decision unit 1 131 included in the image coding 

25 apparatus 1 200 of the second embodiment. 

[0221] The image coding apparatus 1200a includes a decoding cycle multiplier data generator (second data genera- 
tor) 1233a which generates the decoding cycle multiplier data DEm indicating the frame decoding cycle by a multiplier 
M for the sub-unit time (1/N), based on the input image signal Sg, in place of the display cycle multiplier data generator 
1233 of the image coding apparatus 1200. The apparatus 1200a further includes a decoding time data generator (third 

30 data generator) 1 234a which generates the decoding time data DEyn indicating the decoding time Dh(n) of each frame 
F(n), based on the input image signal Sg, in place of the display time data generator 1234 of the image coding appara- 
tus 1200. 

[0222] In the image coding apparatus 1200a, the multiplexer (MUX) 1220a multiplexes the sub-unit time data Dk, the 
decoding cycle multiplier data DEm. and the decoding time data DEty with the coded image data Cgn of each frame 
35 F(n), and outputs a coded image signal having a fixed decoding cycle or a coded image signal having a variable decod- 
ing cycle as a multiplexed bit stream M2a. 

[0223] Other constituents of the apparatus 1200a are identical to those of the image coding apparatus 1200 of the 
second embodiment 

[0224] Hereinafter, the operation of the image coding apparatus 1200a according to the modification of the second 

40 embodiment will be briefly described. 

[0225] When the image signal Sg is input to the apparatus 1200a, the decision unit 1 131a decides whether the decod- 
ing cycle of the image signal Sg is variable or not, and outputs the decoding cycle identifier DEf indicating the result of 
the decision. Meanwhile, the first data generator 1232a generates the sub-unit time data Dk, and the second and third 
data generators 1233a and 1234a generate the decoding cycle multiplier data DEm and the decoding time data DEyn, 

45 respectively. The encoder 1110 encodes the image signal Sg and outputs coded image data Cgn. 

[0226] The decoding cycle identifier DEf from the decision unit 1231a, the coded data Cgn from the encoder 1110, 
and the data Dk and DEyn from the first and the third data generators 1232 and 1234a are input to the multiplexer 
1 220a. Further, the decoding cycle multiplier data DEm from the second data generator 1 233a is input to the multiplexer 
1220a through the switch 1241a. The multiplexer 1220a multiplexes these data, and outputs a coded image signal hav- 

so ing a fixed decoding cycle or a variable decoding cycle as the multiplexed bit stream M2a. 

[0227] On the other hand, the process for decoding the coded image signal having the data structure according to 
this modification of the second embodiment is realized by replacing steps S41, S42a. S43b. S44a. S44b. S45a, and 
S47a in the flow of figure 10 as follows. 

[0228] That is, the process of deciding the display cycle in step S41 is replaced with the process of deciding whether 
55 the decoding cyde is fixed or not; the process of reading the display cycle multiplier data Dm in step S42a is replaced 
with the process of reading the decoding cycle multiplier data DEm; and the process of reading the data Dyn indicating 
the display time h(n) in step S43b is replaced with the process of reading the data DEyn indicating the decoding time 
Dh(n). 
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[0229] Further, the process of obtaining the display time h(n) based on the data Dyn in step S44b is replaced with the 
process of obtaining the decoding time Dh(n) based on the data DEyn; and the process of setting the count value n' to 
0 in step S44a, which count value corresponds to the number n' of each frame F'(n') counted in the display order from 
the head frame, is replaced with the process of setting the count value n to 0, which count value corresponds to the 

5 number n of each frame F(n) counted in the transmission order from the head frame. 

[0230] Further, the process of obtaining the display time h'(n') of each frame F(ri) by rV(n')=n'xT in step S45a is 
replaced with the process of obtaining the decoding time Dh(n) of each frame F(n) by Dh(n)=nxDT based on the 
decoding cycle DT and the number n indicating the transmission order of the frame; and the process of incrementing 
the count value n' in step S47a is replaced with incrementing the count value n. 

io [0231] Figure 1 1 (b) is a block diagram illustrating the structure of an image decoding apparatus 2200a as hardware 
performing the decoding process according to the modification of the second embodiment. 

[0232] The image decoding apparatus 2200a receives the multiplexed bit stream M2a output from trie image coding 
apparatus 1200a, and subjects the multiplexed bit stream M2a to reproduction including decoding and display. 
[0233] More specifically, the image decoding apparatus 2200a includes a demultiplexer (DEMUX) 2210a for extract- 

15 ing, from the multiplexed bit stream M2a, the sub-unit time data Dk, the decoding cycle identifier DEf, the decoding 
cycle multiplier data DEm, the decoding time data DEyn, and the coded image data Cgn, and outputting these data, in 
place of the demultiplexer 2210 of the image decoding apparatus 2200 of the second embodiment 
[0234] Further, the image decoding apparatus 2200a includes, in place of the first and second ON/OFF switches 2240 
and 2250 of the image decoding apparatus 2200, a first ON/OFF switch 2240a for switching the circuit between the ON 

20 state where the decoding cycle multiplier data DEm is transmitted and the OFF state where the data DEm is cut off, 
based on the decoding cycle identifier DEf; and a second ON/OFF switch 2250a for switching the circuit between the 
ON state where the decoding time data DEyn is transmitted and the OFF state where the data DEyn is cut off, based 
~, on the decoding cycle identifier DEf. 
[0235] In the image decoding apparatus 2200a, the sub-unit time data Dk from the demultiplexer 2210a, and the 

25 decoding cycle multiplier data DEm and the decoding time data DEyn from the first and second ON/OFF switches 
2240a and 2250a are supplied to the decoder 2220a and the display unit 2230a. 

[0236] In the decoder 2220a. the coded image data Cgn of each frame F(n), the decoding cycle of which is fixed, is 
decoded frame by frame at the timing (decoding time Dh(n)=DTxn ) decided by the sub-unit time data Dk and the 
decoding cycle multiplier data DEm, while the coded image data Cgn of each frame F(n), the decoding cycle of which 

30 is variable, is decoded frame by frame at the timing (decoding time Dh(n)) decided by the decoding time data DEyn. 
[0237] Further, in the display unit 2230a, the decoded image data Rg of each frame F(n), the decoding cycle of which 
is fixed, is displayed frame by frame at the timing (display time h(n)) decided by the sub-unit time data Dk and the decod- 
ing cycle multiplier data DEm, while the decoded image data Rg of each frame F(n), the decoding cycle of which is var- 
iable, is displayed frame by frame at the timing (display time h(n)) decided by the decoding time data DEyn. 

35 [0238] Other constituents of the image decoding apparatus 2200a are identical to those of the image decoding appa- 
ratus 2200 of the second embodiment. 

[0239] Hereinafter, the operation of the image decoding apparatus 2200a will be described briefly. 

[0240] When the multiplexed bit stream M2a is input to the apparatus 2200a, in the demultiplexer 221 0a. the sub-unit 

time data Dk, the decoding cycle multiplier data DEm, the decoding cycle identifier DEf, the decoding time data DEyn, 

40 and the coded image data Cgn are separated from the bit stream M2a. 

[0241] In the decoder 2220a, when the decoding cycle of the input coded image signal is fixed, the coded image data 
Cgn is decoded frame by frame at the timing decided by the sub-unit time data Dk and the decoding cycle multiplier 
data DEm. When the decoding cycle of the input decoded image signal is variable, the coded image data Cgn is 
decoded frame by frame at the timing (decoding time Dh(n)) decided by the decoding time data DEyn. The decoding 

45 time of the coded image signal, the decoding cycle of which is fixed, is decided by the product of the number n indicating 
the transmission order and the decoding cycle DT (=(1/N)xM) , and the decoding time of the coded image signal, the 
decoding cycle of which is variable, is decided by the decoding cycle data DEyn. 

[0242] In the display unit 2230a. the image of each frame F(n) corresponding to the decoded image data Rg, the 
decoding cycle of which is fixed, is displayed at a prescribed timing based on the sub-unit time data Dk and the display 

so cycle multiplier data DEm. On the other hand, the image of each frame F(n) corresponding to the decoded image data 
Rg, the decoding cycle of which is variable, is displayed at a prescribed timing based on the decoding time data DEty. 
[0243] In this modification of the second embodiment, as in the second embodiment, the coded image signal Cgn 
obtained by coding an image signal includes the decoding cycle identifier DEf indicating whether the cycle of image 
decoding for each frame is variable or not. the sub-unit time data Dk and the decoding cycle multiplier data DEm which 

55 indicate a fixed decoding cycle, and the decoding time data DEyn which indicates the decoding time. Therefore, when 
the cycle of image decoding for each frame is fixed, the coded image signal can be decoded by a simple circuit struc- 
ture, i.e., based on the sub-unit time data Dk corresponding to one image and the decoding cycle multiplier data DEm 
having a relatively small data quantity (bit number), without referring to the decoding time data DEyn having a relatively 
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large data quantity (bit number) for each frame. 

[0244] Further, when the cycle of image decoding for each frame is variable, as in the conventional decoding process, 
the coded image signal can be decoded with reference to the decoding time data DEyn for each frame. 
[0245] In the modification of the second embodiment, emphasis has been placed on an image decoding apparatus 
5 2200a which performs image display of each frame as well as decoding of each frame according to data for deciding 
the decoding timing of each frame, which data is included in the coded image signal. However, an image decoding 
apparatus which performs decoding of each frame as well as image display of each frame according to data for deciding 
the display timing of each frame, which data is included in the coded image signal, is also within the scope of the 
present invention. 

w [0246] In this case, the decoding timing at which decoding of each frame is carried out is set according to display tim- 
ing data of plural frames including an object frame to be decoded. That is, based on the display timing data of the object 
frame and the display timing data of the next frame transmitted subsequently to the object frame, the decoding timing 
of the object frame is set at a timing that is earlier, by a prescribed offset time, than the earlier display timing between 
the display timings of the object frame and the next frame. 

is [0247] To be specific, when the display timing of the object frame is earlier than the display timing of the next frame 
transmitted subsequently to the object frame, the offset time is set to a length longer than the time required for decoding 
the object frame. On the other hand, when the display timing of the next frame (for example, B-VOP) transmitted sub- 
sequently to the object frame (for example, P-VOP) is earlier than the display timing of the object frame, the offset time 
is set to a length longer than the sum of the time required for decoding the object frame and the time required for decod- 

20 ing the next frame. 

[0248] When a coding program or a decoding program for implementing, with software, the image processing by the 
coding apparatus or the decoding apparatus according to any of the first and second embodiments and the modifica- 
tions thereof, is recorded in data storage media such as a floppy disk, the image processing can be easily executed in 
an independent computer system. 
25 [0249] Figures 12(a)-12(c) are diagrams for explaining the case where the coding process or the decoding process 
according to any of the embodiments and the modifications of the invention is executed by a computer system using a 
floppy disk which contains the coding program or the decoding program. 

[0250] Figure 12(a) shows a front view of a floppy disk FD, a cross-sectional view thereof, and a floppy disk body D. 
Figure 12(b) shows an example of a physical format of the floppy disk body D. 

30 [0251] The f loppy disk body D is contained in a case PC, providing the floppy disk FD. On the surface of the disk body 
D, a plurality of tracks Tr are formed concentrically from the outer circumference of the disk toward the inner circumfer- 
ence. Each track is divided into 16 sectors (Se) in the angular direction. Therefore, in the floppy disk FD containing the 
above-mentioned program, data of the program are recorded in the assigned sectors on the floppy disk body D. 
[0252] Figure 12(c) shows the structure for recording the program in the floppy disk FD and performing the image 

35 processing by software using the program stored in the floppy disk FD. 

[0253] To be specific, when the program is recorded in the floppy disk FD, data of the program are written in the floppy 
disk FD from the computer system Cs through the floppy disk drive FDD. When the above-mentioned image coding 
apparatus or image decoding apparatus is constructed in the computer system Cs by the program recorded in the 
floppy disk FD, the program is read from the floppy disk FD by the floppy disk drive FDD and then loaded to the com- 

40 puter system Cs. 

[0254] Although in the above description a floppy disk is employed as a data storage medium, an optical disk may be 
employed. Also in this case, the coding process or the decoding process can be performed by software in similar man- 
ner to the case of using the floppy disk. The data storage medium is not restricted to the floppy disk and the optical disk, 
and any medium may be employed as long as it can contain the program, for example, an IC card or a ROM cassette. 
45 [0255] Furthermore, assuming that a coded image signal stored in a data storage medium such as a floppy disk has 
a data structure according to any of the first and second embodiments and the modifications thereof, when the coded 
image signal from the floppy disk is decoded and the image corresponding to the decoded image signal is displayed, it 
is possible to perform reproduction of the coded image signal, including decoding and image display of the image sig- 
nal, with a simple circuit structure if the frame display cycle or the frame decoding cycle is fixed. 

so 

Claims 

1 . A data structure of an image signal for which image reproduction of each frame is performed at a prescribed cycle, 
said image signal data structure including a reproduction cycle identifier indicating whether the cycle of image 

55 reproduction for each frame is variable or not. 

2. The image signal data structure of Claim 1 wherein said reproduction cycle identifier is a display cycle identifier 
indicating whether the cycle of image display for each frame is variable or not. 
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3. The image signal data structure of Claim 2 including: 

a fixed display cycle identifier indicating that the cycle of image display for each frame is fixed, as said display 
cycle identifier; 

5 display cycle data indicating the cycle of image display for each frame; and 

frame position data corresponding to each frame and indicating the positional relationship of each frame with 
the previous and subsequent frames. 

4. The image signal data structure of Claim 2 including: 

10 

a variable display cycle identifier indicating that the cycle of image display for each frame is variable, as said 
display cycle identifier; and 

display timing data indicating the timing at which image display of each frame is performed, said timing being 
set relatively to a desired reference time selected from at least one reference time according to each frame. 

15 

5. The image signal data structure of Claim 1 wherein said reproduction cycle identifier is a decoding cycle identifier 
indicating whether the cycle of decoding of a coded image signal corresponding to each frame is variable or not. 

6. The image signal data structure of Claim 5 including: 

a fixed decoding cycle identifier indicating that the cycle of decoding for each frame is fixed, as said decoding 
cycle identifier; 

decoding cycle data indicating the cycle of decoding for each frame; and 

frame position data corresponding to each frame and indicating the positional relationship of each frame with 
the previous and subsequent frames. 

7. The image signal data structure of Claim 5 including: 

a variable decoding cycle identifier indicating that the cycle of decoding for each frame is variable, as said 
30 decoding cycle identifier; and 

decoding timing data indicating the timing at which decoding of each frame is performed, said timing being set 
relatively to a desired reference time selected from at least one reference time, according to each frame. 

8. An image coding method for coding an image signal corresponding to a specific image to generate a coded image 
35 signal, and outputting the coded image signal together with a reproduction cycle identifier indicating whether the 

cycle of image reproduction for each of frames constituting the image is variable or not, wherein: 

when an image signal having a fixed cycle of image reproduction for each frame is input as said image signal 
to be coded, reproduction cycle data and frame position data are output together with the reproduction cycle 
*o identifier, said reproduction cycle data indicating the cycle of image reproduction for each frame, and said 

frame position data corresponding to each frame and indicating the positional relationship of each frame with 
the previous and subsequent frames; and 

when an image signal having a variable cycle of image reproduction for each frame is input as said image sig- 
nal to be coded, reproduction timing data is output together with the reproduction cycle identifier, said repro- 
45 duction timing data indicating the timing at which image reproduction of each frame is performed, said timing 

being set relatively to a desired reference time selected from at least one reference time, according to each 
frame. 

9. The image coding method of Claim 8 wherein: 

50 

said reproduction cycle identifier is a display cycle identifier indicating whether the cycle of image display for 
each frame is variable or not; 

said reproduction cycle data is display cycle data indicating the cycle of image display for each frame; and 
said reproduction timing data is display timing data indicating the timing at which image display of each frame 
55 is performed, said timing being set relatively to a desired reference time selected from at least one reference 

time, according to each frame. 

10. The image coding method of Claim 8 wherein: 
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said reproduction cycle identifier is a decoding cycle identifier indicating whether the cycle of decoding of 
coded image data corresponding to each frame is variable or not; 

said reproduction cycle data is decoding cycle data indicating the cycle of decoding for each frame; and 
said reproduction timing data is decoding timing data indicating the timing at which decoding of each frame is 
5 performed, said timing being set relatively to a desired reference time selected from at least one reference 

time, according to each frame. 

1 1 . An image decoding method for decoding a coded image signal which includes coded image data corresponding to 
frames constituting an image, and a reproduction cycle identifier indicating whether the cycle of image reproduction 

io for each frame is variable or not, wherein: 

when the reproduction cycle identifier indicates that the cycle of image reproduction for each frame is fixed, 
decoded image data obtained by decoding the coded image data corresponding to each frame becomes 
image data having a reproduction timing which is decided according to reproduction cycle data indicating the 

75 cycle of image reproduction for each frame and frame position data indicating the positional relationship of 

each frame with the previous and subsequent frames, which data are included in the coded image signal; and 
when the reproduction cycle identifier indicates that the cycle of image reproduction for each frame is variable, 
decoded image data obtained by decoding the coded image data corresponding to each frame is image data 
having a reproduction timing decided according to reproduction timing data which is included in the coded 

20 image signal and indicates the timing at which image reproduction of each frame is carried out, said timing 

being set relatively to a desired reference time selected from at least one reference time, according to each 
frame. 

1 2. The image decoding method of Claim 1 1 wherein: 

25 

said reproduction cycle identifier is a display cycle identifier indicating whether the cycle of image display for 
each frame is variable or not; 

said reproduction cycle data is display cycle data indicating the cycle of image display for each frame; and 
said reproduction timing data is display timing data indicating the timing at which image display of each frame 
30 is performed. 

1 3. The image decoding method of Claim 1 1 wherein a decoding timing at which decoding of each frame is carried out 
is set according to display timing data of plural frames including an object frame to be decoded. 

35 1 4. The image decoding method of Claim 1 3 wherein, based on the display timing data of the object frame and the dis- 
play timing data of the next frame transmitted subsequently to the object frame, the decoding timing of the object 
frame is set at a timing that is earlier, by a prescribed offset time, than the earlier display timing between the display 
timings of the object frame and the next frame. 

40 15. The image decoding method of Claim 14 wherein: 

when the display timing of the object frame is earlier than the display timing of the next frame, the offset time 
is set to a length longer than the time required for decoding the object frame; and 

when the display timing of the next frame is earlier than the display timing of the object frame, the offset time 
45' is set to a length longer than the sum of the time required for decoding the object frame and the time required 

for decoding the next frame. 

16. The image decoding method of Claim 1 1 wherein: 

so said reproduction cycle identifier is a decoding cycle identifier indicating whether the cycle of decoding of 

coded image data corresponding to each frame is variable or not; 

said reproduction cycle data is decoding cycle data indicating the cycle of decoding for each frame; and 
said reproduction timing data is decoding timing data indicating the timing at which decoding of each frame is 
performed. 

55 

17. The image signal data structure of Claim 1 further including: 

sub-unit time data indicating the length of a sub-unit time which is obtained by dividing a prescribed time inter- 
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val into N (natural number), by the natural number N; and 

reproduction cycle multiplier data indicating that the cycle of image reproduction for each frame is the sub-unit 
time multiplied by M (natural number), by the multiplier M 

5 18. The image signal data structure of Claim 1 7 wherein: 

said reproduction cycle identifier is a display cycle identifier indicating whether the cycle of image display for 
each frame is variable or not; and 

said reproduction cycle multiplier data is display cycle multiplier data which indicates that the cycle of image 
10 display for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. 

1 9. The image signal data structure of Claim 1 7 wherein: 

said reproduction cycle identifier is a decoding cycle identifier indicating whether the cycle of decoding for each 
'5 frame is variable or not; and 

said reproduction cycle multiplier data is decoding cycle multiplier data which indicates that the cycle of decod- 
ing for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. 

20. An image coding method for coding an image signal corresponding to a specific image to generate a coded image 
signal, and outputting the coded image signal together with a reproduction cycle identifier indicating whether the 
cycle of image reproduction for each of frames constituting the image is variable or not, wherein: 

when an image signal having a fixed cycle of image reproduction for each frame is input as said image signal 
to be coded, sub-unit time data and reproduction cycle multiplier data are output together with the reproduction 
cycle identifier, said sub-unit time data indicating the length of a sub-unit time which is obtained by dividing a 
prescribed time interval into N (natural number), by the natural number N, and said reproduction cycle multi- 
plier data indicating that the cycle of image reproduction for each frame is the sub-unit time multiplied by M 
(natural number), by the multiplier M. 

30 21. The image coding method of Claim 20 wherein: 

said reproduction cycle identifier is a display cycle identifier indicating whether the cycle of image display for 
each frame is variable or not; and 

said reproduction cycle multiplier data is display cycle multiplier data which indicates that the cycle of image 
35 display for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. 

22. The image coding method of Claim 20 wherein: 

said reproduction cycle identifier is a decoding cycle identifier indicating whether the cycle of decoding for each 
4 ° frame is variable or not; and 

said reproduction cycle multiplier data is decoding cycle multiplier data which indicates that the cycle of decod- 
ing for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. 

23. An image decoding method for decoding a coded image signal including the following data: coded image data cor- 
responding to frames constituting an image; a reproduction cycle identifier indicating whether the cycle of image 
reproduction for each frame is variable or not; sub-unit time data indicating the length of a sub-unit time which is 
obtained by dividing a prescribed time interval into N (natural number), by the natural number N; and reproduction 
cycle multiplier data indicating that the cycle of image reproduction for each frame is the sub-unit time multiplied by 
M (natural number), by the multiplier M, wherein: 
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when the reproduction cycle identifier indicates that the cycle of image reproduction for each frame is fixed, 
decoded image data obtained by decoding the coded image data corresponding to each frame makes image 
data having a reproduction timing which is decided according to the sub-unit time data indicating the length of 
the sub-unit time and the reproduction cycle multiplier data indicating the cycle of image reproduction for each 
55 frame. 

24. The image decoding method of Claim 23 wherein: 
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said reproduction cycle identifier is a display cycle identifier indicating whether the cycle of image display for 
each frame is variable or not; and 

said reproduction cycle multiplier data is display cycle multiplier data which indicates that the cycle of image 
display for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. 

25. The image decoding method of Claim 23 wherein: 

said reproduction cycle identifier is a decoding cycle identifier indicating whether the cycle of decoding for each 
frame is variable or not; and 

said reproduction cycle multiplier data is decoding cycle multiplier data which indicates that the cycle of decod- 
ing for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M. 

26. An image coding apparatus for coding an image signal corresponding to a specific image, comprising: 

an encoder for coding an input image signal and outputting coded image data; 

a cycle decision unit for deciding whether the cycle of image reproduction for each of frames constituting the 
image is variable or not, based on the image signal, and outputting a reproduction cycle identifier indicating the 
result of the decision; 

a first data generator for generating reproduction cycle data indicating the cycle of image reproduction for each 
frame, according to the image signal; 

a second data generator for generating frame position data corresponding to each frame and indicating the 
positional relationship of each frame with the previous and subsequent frames, according to the image signal; 
a third data generator for generating reproduction timing data indicating the timing at which image reproduction 
for each frame is carried out, according to the image signal; 

an ON/OFF switch for switching the circuit between the ON state where the reproduction cycle data is trans- 
mitted and the OFF state where the reproduction cycle data is cut off. according to the reproduction cycle iden- 
tifier; 

a selector switch for selecting one of the frame position data and the reproduction timing data, according to the 
reproduction cycle identifier; 

a multiplexer for multiplexing the outputs from the encoder, the cycle decision unit, and the switches in a pre- 
scribed order; and 

said image coding apparatus outputting a bit stream obtained by the multiplication as a coded image signal. 

27. An image decoding apparatus for decoding and reproducing the coded image signal output from the image coding 
apparatus of Claim 26, comprising: 

a demultiplexer for receiving the coded image signal, and separating the coded image signal into the coded 
image data, the reproduction cycle identifier, the reproduction cycle data, the frame position data, and the 
reproduction timing data, and then outputting these data; 

a decoder for decoding the coded image data, frame by frame, to generate decoded image data; 
an ON/OFF switch for switching the circuit between the ON state where the reproduction cycle data is trans- 
mitted and the OFF state where the reproduction cycle data is cut off. according to the reproduction cycle iden- 
tifier; 

a selector switch for selecting one of the frame position data and the reproduction timing data, according to the 
reproduction cycle identifier; and 

a display unit for performing image display of each frame according to the decoded image data; 

wherein at least one of the decoding by the decoder and the image display by the display unit is per- 
formed at a reproduction timing decided by the reproduction cycle data and the frame position data or a repro- 
duction timing decided by the reproduction timing data, according to the reproduction cycle identifier. 

28. An image coding apparatus for coding an image signal corresponding to a specific image, comprising: 

an encoder for coding an input image signal and outputting coded image data; 

a cycle decision unit for deciding whether the cycle of image reproduction for each of frames constituting the 
image is variable or not. based on the image signal, and outputting a reproduction cycle identifier indicating the 
result of the decision; 

a first data generator for generating sub-unit time data indicating the length of a sub-unit time which is obtained 
by dividing a prescribed time interval into N (natural number), by the natural number N; 
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a second data generator for generating reproduction cycle multiplier data indicating that the cycle of image 
reproduction for each frame is the sub-unit time multiplied by M (natural number), by the multiplier M, accordinq 
to the image signal; 

a third data generator for generating reproduction timing data indicating the timing at which image reproduction 
5 of each frame is carried out, according to the image signal; 

a first ON/OFF switch for switching the circuit between the ON state where the sub-unit time data is transmitted 

and the OFF state where the sub-unit time data is cut off, according to the reproduction cycle identifier; 

a second ON/OFF switch for switching the circuit between the ON state where the reproduction cycle multiplier 

data is transmitted and the OFF state where the reproduction cycle multiplier data is cut off, according to the 
to reproduction cycle identifier; 

a multiplexer for multiplexing the outputs from the encoder, the cycle decision unit, the third data generator, and 

the first and second ON/OFF switches in a prescribed order; and 

said image coding apparatus outputting a bit stream obtained by the multiplication as a coded image signal. 

is 29. An image decoding apparatus for decoding and reproducing the coded image signal output from the image coding 
apparatus of Claim 28, comprising: 

a demultiplexer for receiving the coded image signal, and separating the coded image signal into the coded 
image data, the reproduction cycle identifier, the sub-unit time data, the reproduction cycle multiplier data, and 
20 the reproduction timing data, and then outputting these data; 

a decoder for decoding the coded image data, frame by frame, to generate decoded image data; 
a first ON/OFF switch for switching the circuit between the ON state where the reproduction cycle multiplier 
, . _ data is transmitted and the OFF state where this data is cut off, according to the reproduction cycle identifier; 

a second ON/OFF switch for switching the circuit between the ON state where the reproduction timing data is 
transmitted and the OFF state where this data is cut off, according to the reproduction cycle identifier; 
a display unit for performing image display of each frame according to the decoded image data; 

wherein at least one of the decoding by the decoder and the image display by the display unit is per- 
formed at a reproduction timing decided by the sub-unit time data and the reproduction cycle multiplier data or 
a reproduction timing decided by the reproduction timing data, according to the reproduction cycle identifier. 
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30. A data storage medium containing an image signal for which image reproduction for each frame is performed at a 
prescribed cycle, said image signal including a reproduction cycle identifier indicating whether the cycle of image 
reproduction for each frame is variable or not. 

35 31. A data storage medium containing an image signal for which image reproduction for each frame is performed at a 
prescribed cycle, said image signal including: 

a reproduction cycle identifier indicating whether image reproduction for each frame is variable or not; 
sub-unit time data indicating the length of a sub-unit time which is obtained by dividing a prescribed time inter- 
40 v al into N (natural number), by the natural number N; and 

reproduction cycle multiplier data indicating that the cycle of image reproduction for each frame is the sub-unit 
time multiplied by M (natural number), by the multiplier M. 
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32. A data storage medium containing an image processing program, said image processing program being a coding 
program which enables a computer to execute the coding of an image signal by the image coding method of Claim 

8. 



33. A data storage medium containing an image processing program, said image processing program being a decod- 
ing program which enables a computer to execute the decoding of a coded image signal by the imaqe decodinq 

so method of Claim 11. ~ 

34. A data storage medium containing an image processing program, said image processing program being a coding 
program which enables a computer to execute the coding of an image signal by the image coding method of Claim 
20. 

55 

35. A data storage medium containing an image processing program, said image processing program being a decod- 
mg program which enables a computer to execute the decoding of a coded image signal by the image decoding 
method of Claim 23. 
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Fig.4 (a) 
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Fig.5 
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Fig.6 (a) 
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Fig.9 (a) 
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Fig. 10 



^ START ) 



Read sub-unit time data Dk 



S41 



S42b 



S46b 

s 



Set counter value n to 0 

i - 




S40 



No Display 

cycle is fixed ? 



Read display time 
multiplier data Dm 



) 



S42a 



S43b 



S44b 



Read display time data Dyn 

I 



Obtain T=1/N • M 
based on Dk and Dm 



i . 



S43a 



Obtain display time h(n) 
based on Dyn 



n n+1 
5 



Set counter value n' to 0 



r S44a 



S45a 



Obtain display time 
h'(n') =n'XT 



n S47a 

( 

) 

n f — n'+1 




( E " D ) 



38 



EP 0 914 008 A2 



Fig.11 (a) 
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Fig.12 (a) 
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Fig. 13 Prior Art 
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Fig.17(a) 
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(54) Image signal data structure, image coding method, and image decoding method 



(57) A data structure of an image signal includes a 
reproduction cycle identifier indicating whether the cycle 
of image display for each frame is variable or not. When 
the display cycle identifier indicates that the display 
cycle is fixed, display cycle data is inserted in a header 
of the coded image data, and data relating to the frame 
number is inserted in each frame. On the other hand, 
when the display cycle identifier indicates that the dis- 
play cycle is variable, display time data is inserted for 
each frame. Therefore, when decoding and displaying 



the coded image data having a fixed display cycle, the 
decoded image data can be displayed by a simple cir- 
cuit structure, i.e., based on the display cycle data and 
the frame number data having a relatively small data 
quantity (bit number), without referring to the display 
time data having a relatively large data quantity for each 
frame. Furthermore, this data structure can be applied 
to a coded image signal having a variable display cycle. 
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Fig.1 (b) 
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